Layer Two Over Multiple Sites

ABSTRACT

An apparatus including a service network and a plurality of Layer 2 sites connected by the service network via a plurality of gateways is provided. The gateways are configured to map a plurality of Internet Protocol (IP) addresses of a plurality of hosts under a plurality of virtual local area networks (VLANs) in a plurality of Layer 2 sites to a plurality of addresses (e.g., MAC or others) of the corresponding other gateways, inform the other gateways in other Layer 2 sites of the IP addresses mapped under each of the VLANs in the local Layer 2 sites, and forward data frames originated from the hosts in the local Layer 2 sites to the other gateways in the other Layer 2 sites when destinations of the data frames are residing in the other Layer 2 sites.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a divisional of co-pending U.S. patentapplication Ser. No. 13/172,554, filed Jun. 29, 2011, by Linda Dunbar,et al., and entitled “Layer Two Over Multiple Sites,” now U.S. Pat. No.9,014,054, which claims the benefit of U.S. Provisional PatentApplication Nos. 61/449,918 filed Mar. 7, 2011, by Linda Dunbar, et al.,and entitled “Directory Server Assisted Address Resolution,” 61/374,514filed Aug. 17, 2010, by Linda Dunbar, et al., and entitled “DelegateGateways and Proxy for Target hosts in Large Layer Two and AddressResolution with Duplicated Internet Protocol Addresses,” 61/359,736filed Jun. 29, 2010, by Linda Dunbar, et al., and entitled “Layer 2 tolayer 2 Over Multiple Address Domains,” and 61/411,324 filed Nov. 8,2010 by Linda Dunbar, et al., and entitled “Asymmetric Network AddressEncapsulation,” each of which is incorporated herein by reference as ifreproduced in its entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

REFERENCE TO A MICROFICHE APPENDIX

Not applicable.

BACKGROUND

Modern communications and data networks are comprised of nodes thattransport data through the network. The nodes may include routers,switches, bridges, or combinations thereof that transport the individualdata packets or frames through the network. Some networks may offer dataservices that forward data frames from one node to another node acrossthe network without using pre-configured routes on intermediate nodes.Other networks may forward the data frames from one node to another nodeacross the network along pre-configured or pre-established paths.

SUMMARY

In one embodiment, the disclosure provides an apparatus including aservice network and a plurality of Layer 2 sites connected by theservice network via a plurality of gateways. The gateways are configuredto map a plurality of Internet Protocol (IP) addresses of a plurality ofhosts under a plurality of virtual local area networks (VLANs) in aplurality of Layer 2 sites to a plurality of addresses (e.g., MAC orothers) of corresponding other gateways, inform the other gateways inother Layer 2 sites of the IP addresses mapped under each of the VLANsin the local Layer 2 sites, and forward data frames originated from thehosts in the local Layer 2 sites to the other gateways in the otherLayer 2 sites when destinations of the data frames are residing in theother Layer 2 sites.

In another embodiment, the disclosure provides a network componentincluding a receiver configured to receive via a service network a frameencapsulated using one or two layers of Ethernet headers that comprise adestination address in an outer Ethernet header destined to a localgateway of a local Layer 2 site from a remote host in a remote Layer 2site and a type field that indicates whether the outer Ethernet headerrequires decapsulation or another type field in the Ethernet header thatindicates whether the destination address in the Ethernet headerrequires translation, a logic circuit configured to translate thedestination address in the Ethernet header to a Media Access Control(MAC) address of a local host based on an Ethernet payload's InternetProtocol (IP) destination address of the local host in the local Layer 2site when the destination address requires translation, and atransmitter configured to send to another gateway of another Layer 2site a frame originated from a local host in the local Layer 2 siteconnected to the other Layer 2 site via the service network when thedestination host resides in the other Layer 2 site.

These and other features will be more clearly understood from thefollowing detailed description taken in conjunction with theaccompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of this disclosure, reference is nowmade to the following brief description, taken in connection with theaccompanying drawings and detailed description, wherein like referencenumerals represent like parts.

FIG. 1 is a schematic diagram of an embodiment of Virtual Private LocalArea Network (LAN) Service (VPLS) interconnected LANs.

FIG. 2 is a schematic diagram of an embodiment of a virtual Layer 2network.

FIG. 3 is a schematic diagram of an embodiment of a border controlmechanism.

FIG. 4 is a schematic diagram of an embodiment of a data frameforwarding scheme.

FIG. 5 is a schematic diagram of another embodiment of a data frameforwarding scheme.

FIG. 6 is a schematic diagram of another embodiment of a data frameforwarding scheme.

FIG. 7 is a schematic diagram of an embodiment of interconnected Layer 2sites.

FIG. 8 is a schematic diagram of an embodiment of a Layer 2 extensionover multiple address domains.

FIG. 9 is a schematic diagram of an embodiment of pseudo Layer 2networks over multiple address domains.

FIG. 10 is a schematic diagram of an embodiment of a domain addressrestriction mechanism.

FIG. 11 is a schematic diagram of another embodiment of a data frameforwarding scheme.

FIG. 12 is a schematic diagram of another embodiment of a data frameforwarding scheme.

FIG. 13 is a schematic diagram of another embodiment of a data frameforwarding scheme.

FIG. 14 is a schematic diagram of another embodiment of a data frameforwarding scheme.

FIG. 15 is a schematic diagram of an embodiment of a broadcast scheme.

FIG. 16 is a schematic diagram of another embodiment of a broadcastscheme.

FIG. 17 is a schematic diagram of an embodiment of interconnectednetwork districts.

FIG. 18 is a schematic diagram of another embodiment of interconnectednetwork districts.

FIG. 19 is a schematic diagram of an embodiment of an Address ResolutionProtocol (ARP) proxy scheme.

FIG. 20 is a schematic diagram of another embodiment of a data frameforwarding scheme.

FIG. 21 is a schematic diagram of another embodiment of an ARP proxyscheme.

FIG. 22 is a schematic diagram of an embodiment of a fail-over scheme.

FIG. 23 is a schematic diagram of an embodiment of a physical server.

FIG. 24 is a schematic diagram of an embodiment of an asymmetric networkaddress encapsulation scheme.

FIG. 25 is a schematic diagram of an embodiment of an ARP processingscheme.

FIG. 26 is a schematic diagram of an embodiment of an extended ARPpayload.

FIG. 27 is a schematic diagram of an embodiment of another data frameforwarding scheme.

FIG. 28 is a protocol diagram of an embodiment of an enhanced ARPprocessing method.

FIG. 29 is a protocol diagram of an embodiment of an extended addressresolution method.

FIG. 30 is a schematic diagram of an embodiment of a network componentunit.

FIG. 31 is a schematic diagram of an embodiment of a general-purposecomputer system.

DETAILED DESCRIPTION

It should be understood at the outset that although an illustrativeimplementation of one or more embodiments are provided below, thedisclosed systems and/or methods may be implemented using any number oftechniques, whether currently known or in existence. The disclosureshould in no way be limited to the illustrative implementations,drawings, and techniques illustrated below, including the exemplarydesigns and implementations illustrated and described herein, but may bemodified within the scope of the appended claims along with their fullscope of equivalents.

Modern data networks, which can be Layer 2 or Layer 3 networks, provideconnectivity to cloud services and virtual machines (VMs) that may needto span across multiple locations or sites. Sometimes, Layer 2 networksfor data centers that connect Clusters of servers (or VMs) and storagedevices have to span across multiple locations. The data center networksmay also need to stay in Layer 2 level to support already deployedapplications and thus save cost, e.g., in millions of dollars. Layer 2communications between the Cluster of servers and/or storage devicesinclude load balancing, database clustering, virtual server failurerecovery, transparent operation below the network layer (Layer 3),spreading a subnet across multiple locations, and redundancy. Layer 2communications also include a keep-alive mechanism between applications.Some applications need the same IP addresses to communicate on multiplelocations, where one server may be Active and another server may be onStandby. The Active and Standby servers (in different locations) mayexchange keep-alive messages between them, which may require a Layer 2keep-alive mechanism.

FIG. 1 illustrates an embodiment of a Virtual Private Local Area Network(LAN) Service (VPLS) interconnected Local Area Networks (LANs) 100. TheVPLS interconnected LANs 100 is a scalable mechanism that can be usedfor connecting Layer 2 networks across multiple data center (DC)locations, e.g., physical locations, to establish a unified or flatLayer 2 network. The VPLS interconnected LANs 100 may comprise a VPLS110 and a plurality of LANs 120 that may be connected to the VPLS 110via a plurality of edge nodes 112, such as edge routers. Each LAN 120may comprise a plurality of Layer 2 switches 122 connected tocorresponding edge nodes 112, a plurality of access switches 124connected to corresponding Layer 2 switches, and a plurality of VMs 126connected to corresponding access switches 124. The components of theVPLS interconnected LANs 100 may be arranged as shown in FIG. 1.

The VPLS 110 may be any network that is configured to connect the LANs120 across different locations or DCs. For instance, the VPLS 110 maycomprise a Layer 3 network to interconnect the LANs 120 across differentDCs. The Layer 2 switches 122 may be configured to communicate at theOpen System Interconnection (OSI) model data link layer. Examples ofdata link protocols include Ethernet for LANs, the Point-to-PointProtocol (PPP), High-Level Data Link Control (HDLC), and Advanced DataCommunication Control Protocol (ADCCP) for point-to-point connections.The access switches 124 may be configured to forward data between theLayer 2 switches 122 and the VMs 126. The VMs 126 may comprise systemvirtual machines that provide system platforms, e.g., operating systems(OSs) and/or process virtual machines that run programs or applications.The VMs 126 in each LAN 120 may be distributed over a plurality ofprocessors, central processor units (CPUs), or computer systems. Aplurality of VMs 126 in a LAN 120 may also share the same systemresources, such as disk space, memory, processor, and/or other computingresources. The VMs 126 may be arranged on a shelf and connected to thecorresponding LANs 120, e.g., via the access switches 124.

Some aspects of the VPLS interconnected LANs 100 may pose impractical orundesirable implementation issues. In one aspect, the VPLS 110 mayrequire implementing a Wide Area Network (WAN) that supports MultipleLabel Protocol Label Switching (MPLS). However, some operators do notsupport MPLS over WAN and thus may have difficulties in implementingVPLS interconnected LANs 100. Further, to resolve host link layeraddresses, e.g., for the VMs 126 across the LANs 120, an IP version four(IPv4) ARP or IP version six (IPv6) Neighbor Discovery (ND) protocol maybe needed, such as the IPv4 ARP described in the Internet EngineeringTask Force (IETF) Request for Comments (RFC) 826 and IPv6 ND describedby IETF RFC 4861, both of which are incorporated herein by reference.The ARP may flood requests to all the interconnected LANs 120 and thusexhaust a substantial amount of system resources (e.g., bandwidth). SuchARP flooding mechanism may suffer from scalability issues, as the numberof LANs 120 and/or VMs 126 increases. The VPLS interconnected LANs 100also need to setup mesh pseudo-wires (PWs) to connect to the LANs 120,which may require intensive configuration and state maintenance oftunnels. In some scenarios, the VPLS 110 may use a Border GatewayProtocol (BGP) to discover a LAN 120 and build a mesh PW for each LAN120.

Optical Transport Virtualization (OTV) is another scalable mechanismthat has been proposed for connecting Layer 2 networks across multiplelocations or DCs to establish a flat Layer 2 network. OTV is a methodproposed by Cisco that depends on IP encapsulation of Layer 2communications. OTV may use an Intermediate System to IntermediateSystem (IS-IS) routing protocol to distribute MAC reachability withineach location (e.g., DC) to other locations. The OTV scheme may alsohave some impractical or undesirable aspects. In one aspect, OTV mayrequire maintaining a relatively large number of multicast groups by aprovider core IP network. Since each LAN may have a separate overlaytopology, there may be a relatively large quantity of overlay topologiesthat are maintained by the service provider IP network, which may pose aburden on the core network. OTV may also require that an edge node touse Internet Group Management Protocol (IGMP) to join differentmulticast groups in the IP domain. If each edge node is connected to aplurality of VLANs, the edge node may need to participate in multipleIGMP groups.

In OTV, edge devices, such as a gateway at each location, may be IPhosts that are one hop away from each other, which may not requireimplementing a link state protocol among the edge devices to exchangereachability information. However, the link state may also be used toauthenticate a peer, which may be needed in OTV if the peer joins a VLANby sending an IGMP version 3 (IGMPv3) report. Alternatively, OTV may usea BGP authentication method. However, the BGP authentication timing maybe different than the IS-IS authentication timing. For example, BGP maybe tuned for seconds performance and IS-IS may be tuned for sub-secondperformance. Further, the IS-IS protocol may not be suitable forhandling a substantially large numbers of hosts and VMs, e.g., tens ofthousands, in each location in the OTV system. OTV may also beunsuitable for supporting tens of thousands of closed user groups.

Disclosed herein are systems and methods for providing a scalablemechanism to connect a plurality of Layer 2 networks at a plurality ofdifferent locations to obtain a flat or single Layer 2 network. Thescalable mechanism may resolve some of the aspects or challenges forobtaining a flat Layer 2 network that spans across multiple locations.The scalable mechanism may facilitate topology discovery across thelocations by supporting scalable address resolution for applications andallowing network switches to maintain a plurality of addressesassociated with all or a plurality of hosts across the locations. Thescalable mechanism may also facilitate forwarding traffic across thedifferent locations and broadcasting traffic, e.g., for unknown hostaddresses, and support multicast groups.

The methods include a border control mechanism to scale a relativelylarge flat Layer 2 over multiple locations. As such, applications,servers, and/or VMs may not be aware of a virtual Layer 2 network thatcomprises multiple Layer 2 networks interconnected by another network,such as a Layer 3, a Layer 2.5, or a Layer 2 network. The Layer 2networks may be located in different or separate physical locations,multiple floors of one location, or multiple rows interconnected byLayer 3. A protocol independent address resolution mechanism may also beused and may be suitable to handle a relatively large virtual Layer 2network and/or a substantially large number of Layer 2 networks overmultiple locations.

FIG. 2 illustrates an embodiment of a virtual Layer 2 network 200 acrossdifferent DC or physical locations. The virtual Layer 2 network 200 maybe a scalable mechanism for connecting Layer 2 networks across multiplelocations, e.g., geographical locations or DCs, or multiple sites withinone data center, to establish a unified or flat Layer 2 network. Thevirtual Layer 2 network 200 may comprise a service network 210 and aplurality of Layer 2 networks 220 that may be connected to the servicenetwork 210 via a plurality of edge nodes 212, such as edge routers. Theservice network 210 may refer herein to an interconnecting network, suchas a service provider network, a core network, a Layer 3 network, aLayer 2 or 2.5 network, or any other network that connects orinterconnects components in multiple sites. Each Layer 2 network 220 maycomprise a plurality of L2GWs 222 connected to corresponding edge nodes212, and a plurality of intermediate switches 224 that may be connectedto the L2GWs 222. The components of virtual Layer 2 network 200 may bearranged as shown in FIG. 2. The intermediate switches 224 may also beconnected to a plurality of hosts and/or VMs (not shown).

The service network 210 may be any network established to interconnectthe Layer 2 networks 220, such as a service provider network. Forexample, the service network 210 may be a Layer 2, Layer 2.5, or Layer 3network, such as a virtual private network (VPN). The service network210 may not be aware of all the addresses, e.g., MAC addresses, behindthe L2GWs 222. The L2GWs 222 may be border nodes in each DC location andhave Layer 2 interfaces to communicate internally in the DC locations.The L2GWs 222 and the intermediate switches 224 may communicate with thehosts and/or VMs in the same locations within the same Layer 2 networks220 using the corresponding MAC addresses of the hosts. However, theL2GWs 222 and the intermediate switches 224 may not need to be aware ofthe MAC addresses of the hosts/VMs in the other Layer 2 networks 220.Instead, a host in one Layer 2 network 220 can use the address of a L2GW222 of another Layer 2 network 220 (in another location or site) as thedestination address to communicate with a target host in the other Layer2 network. When a frame (e.g., an Ethernet frame) arrives at the L2GW222 of the target site, e.g., the other Layer 2 network, the destinationaddress of the target host may be translated by the L2GWs 222 based onthe IP address carried in the payload of the frame, e.g., using anetwork address translation (NAT) table or a MAC address translation(MAT) table, as described below.

In an embodiment, each L2GW 222 may maintain the addresses of all thehosts/VMs within the same Layer 2 network 220 of the L2GW 222 in a localIP addresses information table (Local-IPAddrTable). The L2GW 222 mayalso be configured to implement a proxy ARP function, as describedbelow. Additionally, the L2GW 222 may maintain a MAC forwarding table,which may comprise the MAC addresses for non-IP applications. The MACaddresses may comprise the MAC addresses of the hosts/VMs and theintermediate switches 224 within the same location, e.g., the same Layer2 network 220.

The L2GW 222 may inform its peers (e.g., other L2GWs 222) in otherlocations (e.g., other Layer 2 networks 220) of all the active VLANs andall the IP addresses of the local hosts under each VLAN in its location.If there are non-IP applications within the domain, the L2GW 222 mayalso inform its peers of the MAC addresses and VLANs of those non-IPapplications. A Layer 2 site or Layer 2 network 220 may have many VLANsenabled on the L2GWs' 222 ports and the intermediate switches' 224 portsfor the sake of operation convenience. Thus, a VM or host belonging toany of the enabled VLANs may be moved in without additionalconfiguration. A VLAN that is active in a site (or Layer 2 network 220)may have hosts belonging to this VLAN that resided within this site. TheL2GWs 222 across the different locations may obtain the host IPaddresses of all the other locations, even if the L2GWs 222 may onlykeep the address information for the VLANs that are active in theirlocal sites (e.g., in a remote IP Address Information Table for eachL2GW 222). If there are no hosts in the local domain that belong to VLANidentifier (VID) for the VLAN, then there may be no need to keep theremote hosts information for this VID, since there may be nocommunications for this VID to be targeted to local domain. The termsVLAN and VID are used herein interchangeably to refer to an establishedVLAN, even though a VLAN may be assigned multiple VIDs (e.g., asdescribed in IEEE 802.1Q). Hence, each L2GW 222 may map each group of IPaddresses that belongs to a location (e.g., one of the Layer 2 networks220) to the MAC address of the corresponding L2GW 222 that belongs tothe same location. The L2GW 222 also sends update of the addressinformation to the peers when there is a change in its Local-IPAddrTableto update the information in the other peers. This may allow updatingthe address information and mapping in each L2GW 222 in an incrementalmanner.

FIG. 3 illustrates an embodiment of a border control mechanism 300. Theborder control mechanism 300 may be a scalable mechanism forestablishing a flat or virtual Layer 2 network across multiple sites,locations or DCs. The virtual Layer 2 network may comprise a servicenetwork 310 and a plurality of Layer 2 networks 320 that may beconnected to the service network 310 via a plurality of edge nodes 312,such as edge routers. Each Layer 2 network 220 may comprise a pluralityof L2GWs 322 connected to corresponding edge nodes 312, and a pluralityof intermediate switches 324 that may be connected to the L2GWs 322. Theintermediate switches 324 may also be connected (or reachable) to hosts326, e.g., instantiated on a VM or a server. The components of virtualLayer 2 network may be arranged as shown in FIG. 2 and may be similar tothe corresponding components of the virtual Layer 2 network 200.

Based on the border control mechanism 300, each L2GW 322 may maintainthe IP addresses of hosts in all the locations belonging to the VLANswhich are active in its corresponding local Layer 2 site, e.g.,corresponding to the Layer 2 network 320. Each L2GW 322 may also beaware of the MAC addresses of the peer L2GWs 322 in the other locations.However, the L2GW 322 may not maintain the MAC addresses of the hosts inthe other locations, which may substantially reduce the size of dataexchanged (and stored) among the L2GWs 322, since IP addresses may besummarized (e.g., 10.1.1.x may represent 255 hosts) while MAC addressesmay not be summarized. The IP addresses maintained at the L2GW 322 maybe mapped to the MAC addresses of the corresponding L2GWs 322 of thesame locations. Specifically, each set of host IP addresses that belongto each location or Layer 2 network 300 may be mapped to the MAC addressof the L2GW 322 in that location. However, the L2GWs 322 may exchange,across different locations, a plurality of MAC addresses for nodes thatrun non-IP applications.

To support address resolution across the different locations of thevirtual Layer 2 network, an ARP (or ND) request may be sent from a firsthost 326 (host A) and be intercepted by the corresponding local L2GW 322in a first location or Layer 2 network 320. The host A may send the ARPrequest to obtain the MAC address of a second host 326 (host B) in asecond location or Layer 2 network 320. If the local L2GW 322 has anentry for the host B belonging to the same VLAN as host A, e.g., the IPaddress of the host B, the local L2GW 322 may respond to the ARP/NDrequest by sending its own MAC address to the host A. Alternatively, thelocal L2GW 322 may send the corresponding L2GW's MAC address (where hostB resides) in the ARP/ND response to host A. If the local L2GW 322 doesnot maintain or store an entry for the host B for the VLAN, the localL2GW 322 may assume that the host B does not exist. For example, theL2GWs 322 may update their peers with their local host IP addresses andtheir corresponding VLAN(s) on a regular or periodic basis. It ispossible that some L2GWs 322 may not have received updates for the IPaddresses of newly configured hosts in other locations for some VLANs.In such case, no response is sent back and the requesting entity (hostA) may send multiple ARP/ND requests for the target host.

In an embodiment, the L2GW 222 may send out a plurality of aggregated IPaddresses of the local hosts under each VLAN to the other L2GW 222 inthe other Layer 2 sites. The number of entries in the aggregatedaddresses may be substantially smaller than the corresponding number ofentries in the Local-IPAddrTable of the L2GW 222. in some embodiments,the L2GW 222 may send out requests to all other L2GWs 222 in other Layer2 sites to solicit IP addresses (in aggregated form) under a single VLAN(or any of the VLANs) in the remote sites. This may be useful when ahost belonging to a VLAN that was not active is added to the local siteof the L2GW 222.

Table 1 illustrates an example of mapping host addresses to thecorresponding L2GW's MAC addresses and VLAN according to the bordercontrol mechanism 300. A plurality of L2GW MAC addresses (e.g., L2GW1MAC and L2GW2 MAC) may be mapped to a plurality of corresponding hostaddresses. Each L2GW MAC address may be mapped to a plurality of host IP(or MAC) addresses in a plurality of VLANs (e.g., VLAN#, VLAN-x, . . . )that may be associated with the same location or DC. Each VLAN may alsocomprise a plurality of virtual private groups (VPGs) (or Closed UserGroups) of hosts. A VPG may be a cluster of hosts and/or VMs that belongto a Layer 2 domain (or L2 domain) and may communicate with each othervia Layer 2. A Layer 2 domain may be used herein to refer to asub-location or sub-site in a Layer 2 network. When a Layer 2 networkspans across multiple sites or locations, each site may be referred toherein as a Layer 2 domain. The terms Layer 2 domain, Layer 2 site, andLayer 2 district may be used herein interchangeably. The terms domain,site, and district may also be used herein interchangeably. The hosts inthe VPG may also have multicast groups established among them. Thehosts/VMs within a VPG may span across multiple physical locations.Under many cases, one VLAN may be dedicated to one customer, e.g., theremay be only one VPG per VLAN. As such, there may be no need to have theVPG column (or attribute) in the table under in such cases.

For example, VLAN# may comprise a plurality of hosts in multiple VPGs,including G-x1, G-x2, . . . . And each VPG may comprise a plurality ofhosts. For IP applications, the hosts IP addresses in each VLAN may bemapped to the corresponding L2GW MAC address in the same location, suchas in the case of VLAN# and VLAN-x. IP Addresses may be summarized toreduce the amount of entries in the table. For non-IP applications, thehosts MAC addresses in each VLAN may be mapped to the corresponding L2GWMAC address in the same location for the VLAN, such as in the case ofVLAN-x1. In some cases, there may be only one VPG for each VLAN, andhence the VPG column in Table 1 may not be needed.

TABLE 1 Border Control Mechanism L2GW VLAN VPG Host L2GW1 MAC VLAN# G-x1All IP hosts in this group G-x2 All IP hosts in this group VLAN-x . . .G-xj VLAN-x1 G-j1 MAC (switches and/or nodes without IP addresses) MACG-j2 MAC L2GW2 MAC

FIG. 4 illustrates an embodiment of a data frame forwarding scheme 400that may be used in a virtual Layer 2 network across multiple locationsor DCs. The virtual Layer 2 network may comprise a service network 410and a plurality of Layer 2 networks 420 that may be connected to theservice network 410 via a plurality of edge nodes 412, such as edgerouters. Each Layer 2 network 420 may comprise a plurality of L2GWs 422connected to corresponding edge nodes 412, and a plurality ofintermediate switches 424 that may be connected to the L2GWs 422. Theintermediate switches 424 may also be connected to hosts 426, e.g., VMs.The components of virtual Layer 2 network may be arranged as shown inFIG. 4 and may be similar to the corresponding components of the virtualLayer 2 network 200.

Based on the data frame forwarding scheme 400, the L2GWs 422 may supportthe Institute of Electrical and Electronics Engineers (IEEE) 802.1ahstandard for MAC-in-MAC, which is incorporated herein by reference,using an Ether Type field to indicate that an inner frame needs MACaddress translation. For instance, a first L2GW 422 (GW1) may receive aframe 440, e.g., an Ethernet frame, from a first host 426 (host A) in afirst location (Loc 1). The frame 440 may be intended for a second host426 (host B) in a second location (Loc 2). The frame 440 may comprise aMAC destination address (MAC-DA) 442 for GW1 (L2GW-Loc1), a MAC sourceaddress (MAC-SA) 444 for host A (A's MAC), an IP destination address(IP-DA) 446 for host B (B), an IP source address (IP-SA) 448 for host A(A), and payload. GW1 may then add an outer MAC header to the frame 440to obtain an inner frame 460. The outer MAC header may comprise a MAC-DA462 for GW2 (L2GW-Loc2), a MAC-SA 464 for GW1 (L2GW-Loc1), and an EtherType 466 that indicates that the inner frame 460 needs MAC addresstranslation. The inner frame 460 may also comprise a MAC-DA 468 for GW1(L2GW-Loc1) and a MAC-SA 470 for host A (A's MAC). The inner frame 460may then be forwarded in the service network 410 to GW2, which mayprocess the outer MAC header to translate the MAC addresses of theframe. As such, GW2 may obtain a second frame 480, which may comprise aMAC-DA 482 for host B (B's MAC), a MAC-SA 484 for host A (A's MAC), anIP-DA 486 for host B (B), an IP-SA 488 for host A (A), and payload. Thesecond frame 480 may then be forwarded to host B in Loc 2.

The data frame forwarding scheme 400 may be simpler to implement thanCisco's OTV scheme which requires encapsulating an outer IP header.Additionally, many Ethernet chips support IEEE 802.1ah. A serviceinstance-tag (I-TAG), such as specified in 802.1ah, may be used todifferentiate between different VPGs. Thus, an I-TAG field may also beused in the data frame forwarding scheme 400 to separate betweenmultiple VPGs of the provider domain, e.g., in the service network 410.GW2 may perform the MAC translation scheme described above using a MAT,which may be similar to using a NAT for translating a public IP into aprivate IP. Unlike the NAT scheme that is based on a TransmissionControl Protocol (TCP) session, the MAT scheme may be based on using aninner IP address to find the MAC address.

FIG. 5 illustrates an embodiment of another data frame forwarding scheme500 for non-IP applications. The data frame forwarding scheme 500 mayuse MAC addresses of non-IP hosts or hosts that implement non-IPapplications instead of IP addresses to forward frames between the hostsin different locations in a virtual Layer 2 network. The virtual Layer 2network may comprise a service network 510 and a plurality of Layer 2networks 520 that may be connected to the service network 510 via aplurality of edge nodes 512, such as edge routers. Each Layer 2 network520 may comprise a plurality of L2GWs 522 connected to correspondingedge nodes 512, and a plurality of intermediate switches 524 that may beconnected to the L2GWs 522. The intermediate switches 524 may also beconnected to hosts 526, e.g., VMs. The components of virtual Layer 2network may be arranged as shown in FIG. 5 and may be similar to thecorresponding components of the virtual Layer 2 network 200.

Based on the data frame forwarding scheme 500, the L2GWs 522 may supportIEEE 802.1ah for MAC-in-MAC. For instance, a first L2GW 520 (GW1) mayreceive a frame 540, e.g., an Ethernet frame, from a first host 526(host A) in a first location (Loc 1). The frame 540 may be intended ordestined for a second host 526 (host B) in a second location (Loc 2).The frame 540 may comprise a MAC-DA 542 for GW1 (L2GW-Loc1), a MAC-SA544 for host A (A's MAC), and payload. GW1 may then add outer MAC headerto the frame 540 to obtain an inner frame 560. The outer MAC header maycomprise a MAC-DA 562 for GW2 (L2GW-Loc2), a MAC-SA 564 for GW1(L2GW-Loc1), and an Ether Type 566 that indicates that the inner frame560 is a MAC-in-MAC frame. The inner field 560 may also comprise aMAC-DA 568 for host B (B's MAC) and a MAC-SA 570 for host A (A's MAC).The inner frame 560 may then be forwarded in the service network 510 toGW2, which may process the inner frame 560 to obtain a second frame 580.The second frame 580 may comprise a MAC-DA 582 for host B (B's MAC) anda MAC-SA 584 for host A (A's MAC), and payload. The second frame 580 maythen be forwarded to host B in Loc 2.

The data frame forwarding scheme 500 may be simpler to implement thanCisco's OTV scheme which requires encapsulating outer IP header.Additionally, many Ethernet chips support IEEE 802.1ah. An I-TAG, asdescribed in 802.1ah, may be used to differentiate between differentVPGs. Thus, an I-TAG field may also be used in the data frame forwardingscheme 500 to separate between multiple VPGs of the provider domain,e.g., in the service network 510. GW2 may process the second frame 580,as described above, without performing a MAC translation scheme.

FIG. 6 illustrates an embodiment of another data frame forwarding scheme600 that may be used in a virtual Layer 2 network across multiplelocations. The data frame forwarding scheme 600 may be used to forwardframes from a host that moves from a previous location to a new locationin the virtual Layer 2 network and maintains the same learned MACaddress for a second host. The virtual Layer 2 network may comprise aservice network 610 and a plurality of Layer 2 networks 620 that may beconnected to the service network 610 via a plurality of edge nodes 612,such as edge routers. Each Layer 2 network 620 may comprise a pluralityof L2GWs 622 connected to corresponding edge nodes 612, and a pluralityof intermediate switches 624 that may be connected to the L2GWs 622. Theintermediate switches 624 may also be connected to hosts 626, e.g., VMs.The components of virtual Layer 2 network may be arranged as shown inFIG. 6 and may be similar to the corresponding components of the virtualLayer 2 network 200.

When a first host 626 (host A) moves from a previous location (Loc 1) toa new location (Loc 3), host A may still use the same learned MACaddress for a second host 626 (host B). According to the data frameforwarding scheme 600, a L2GW 622 of Loc 3 (GW3) may support 802.1ahMAC-in-MAC using an Ether Type field to indicate that an inner frameneeds MAC address translation. GW3 may implement a data frame forwardingscheme similar to the data frame forwarding scheme 400 to send data to asecond L2GW 622 of Loc 2 (GW2) using GW2's MAC address in an outer MACheader. Thus, GW2 may decapsulate the outer MAC header and perform MACaddress translation, as described above (for the data frame forwardingscheme 400).

For instance, GW3 may receive a frame 640, e.g., an Ethernet frame, fromhost A after moving to Loc 3. The frame 640 may be intended for host Bin Loc 2. The frame 640 may comprise a MAC-DA 642 for a previous L2GW622 (GW1) of Loc 1 (L2GW-Loc1), a MAC-SA 644 for host A (A's MAC), anIP-DA 646 for host B (B), an IP-SA 648 for host A (A), and payload. GW3may then add an outer MAC header to the frame 640 to obtain an innerframe 660. The outer MAC header may comprise a MAC-DA 662 for GW2(L2GW-Loc2), a MAC-SA 664 for GW1 (L2GW-Loc1), and an Ether Type 666that indicates that the inner frame 660 needs MAC address translation.The inner frame 660 may also comprise a MAC-DA 668 for host B (B's MAC)and a MAC-SA 670 for host A (A's MAC). The inner frame 660 may then beforwarded in the service network 610 to GW2, which may process the outerMAC header to translate the MAC addresses of the frame. As such, GW2 mayobtain a second frame 680, which may comprise a MAC-DA 682 for host B(B's MAC), a MAC-SA 684 for host A (A's MAC), and payload. The secondframe 680 may then be forwarded to host B in Loc 2.

Further, host B may move from Loc 2 to another location, e.g., Loc 4(not shown). If GW2 has learned that host B has moved from Loc 2 to Loc4, then GW2 may use the MAC address of another L2GW 622 in Loc 4 (GW4)as a MAC-DA in an outer MAC header, as described above. If GW2 has notlearned that host B has moved from Loc 2 to Loc 4, then the frame may beforwarded by GW2 without the outer MAC header. As such, the frame may belost, e.g., in the service network 610. The frame may be losttemporarily until the frame is resent by GW2 after host B announces itsnew location to GW2 or Loc 2.

FIG. 7 illustrates an embodiment of interconnected Layer 2 sites (ordistricts) 700 that may implement a similar border control mechanism asthe virtual Layer 2 networks above. The interconnected Layer 2 sites 700may comprise a plurality of L2GWs 722 connected by a plurality of borderor edge nodes 712. The edge nodes, e.g., edge routers, may belong to aservice network, e.g., a Layer 3 network. The interconnected Layer 2sites 700 may also comprise a plurality of intermediate switches 724connected to the L2GWs 722, and a plurality of VMs 726 connected to theintermediate switches 724. The L2GWs 722, intermediate switches 724, andVMs 726 may support multiple subsets that correspond to a plurality ofLayer 2 (L2) address domains. The components of the interconnected Layer2 sites 700 may be arranged as shown in FIG. 7 and may be similar to thecorresponding components of the virtual Layer 2 network 200.

Each L2 address domain may use a border control mechanism, such as theborder control mechanism 300, where the intermediate switches 724 andVMs 726 within each Layer 2 domain may be aware of local MAC addressesbut not the MAC addresses and VLAN for hosts, servers, and/or VMs 726 inthe other L2 address domains. However, the hosts, servers, and/or VMs726 may communicate with each other as in a single flat Layer 2 networkwithout being aware of the different Layer 2 domains. The Layer 2domains may be interconnected to each other via the border or edge nodes712, which may be interconnected over a core network or service providernetwork (not shown). The L2 address domains may be located in one DCsite or at a plurality of geographic sites. The architecture of theinterconnected Layer 2 sites 700 across the multiple sites (locations)may also be referred to herein as a Layer 2 extension over multiplesites interconnected by a service network (Layer 3, 2.5, 2 or othernetworks), pseudo Layer 2 networks over sites interconnected by aservice network, virtual Layer 2, or pseudo Layer 2 networks.

FIG. 8 illustrates one embodiment of a Layer 2 extension 800 overmultiple sites interconnected by a service network. The Layer 2extension 800 may comprise a plurality of L2GWs 822 connected to aplurality of border or edge nodes 812, which may belong to a serviceprovider or core network (not shown). The Layer 2 extension 800 may alsocomprise a plurality of intermediate switches 824 connected to the L2GWs822, and a plurality of hosts/servers/VMs 826 connected to theintermediate switches 824. The intermediate switches 824 andhosts/servers/VMs 826 may be separated or arranged into a plurality ofL2 address domains. For example, one of the L2 sites is indicated by thedashed line circle in FIG. 8. The L2GWs 822, intermediate switches 824,and hosts/servers/VMs 826 may correspond to a Layer 2 network at one ormultiple DC locations. The components of the Layer 2 extension 800 maybe arranged as shown in FIG. 8 and may be similar to the correspondingcomponents of the virtual Layer 2 network 200.

FIG. 9 is a schematic diagram of an embodiment of pseudo Layer 2networks 900 over multiple locations. The pseudo Layer 2 networks 900may be a mechanism for connecting Layer 2 across multiple locations,e.g., geographical locations or DCs, to establish one flat Layer 2network. The pseudo Layer 2 networks 900 may comprise a service provideror core network 910 and a plurality of Layer 2 network domains 920 thatmay be connected to the service provider or core network 910 via aplurality of edge nodes 912, such as edge routers. Each Layer 2 site 920may be located at a different DC site (or floor, or zone) or locationand may comprise a plurality of L2GWs 922 connected to correspondingedge nodes 912, and a plurality of intermediate switches 924 connectedto corresponding L2GWs 922. The intermediate switches 924 may also beconnected to a plurality of hosts/servers/VMs (not shown). Thecomponents of the pseudo Layer 2 networks 900 may be arranged as shownin FIG. 9 and may be similar to the corresponding components of thevirtual Layer 2 network 200.

FIG. 10 illustrates an embodiment of a domain address restrictionmechanism 1000. The domain address restriction mechanism 1000 may beused in pseudo Layer 2 networks over multiple sites to handle addressresolution between the different Layer 2 sites. The pseudo Layer 2networks over multiple sites may comprise a service network 1010 and aplurality of Layer 2 network sites 1020 that may be connected to theservice network 1010 via a plurality of edge nodes 1012. The Layer 2sites 1020 may be located at the same or different DC sites and maycomprise a plurality of L2GWs 1022 connected to corresponding edge nodes1012, and a plurality of intermediate switches 1024 connected tocorresponding L2GWs 1022. The intermediate switches 1024 may also beconnected to a plurality of hosts/servers/VMs 1026. The components ofthe pseudo Layer 2 networks may be arranged as shown in FIG. 10 and maybe similar to the corresponding components of the virtual Layer 2network 200.

Specifically, a MAC address of a L2GW 1022 in one Layer 2 site 1020 maybe used as a proxy for all or a plurality of the hosts outside thislocal site. In a first option (option 1), a MAC address for a local L2GW1022 in the Layer 2 sites 1020 may be used as the proxy for hosts in theother Layer 2 network sites 1020. In this scenario, only addresses oflocal hosts may be learned by the intermediate switches 1024 andhosts/servers/VMs 1026 in the same local Layer 2 sites 1020. The MACaddresses of external L2GWs 1022 in other Layer 2 sites 1020 may not beexposed to the local Layer 2 sites 1020.

Alternatively, in a second option (option 2), the MAC addresses of L2GWs1022 in a remote Layer 2 site 1020 may be used as a proxy for all hostsresiding in the corresponding site. Under this option, the MAC addressesof external L2GWs 1022 in other Layer 2 sites 1020 may be learned ineach Layer 2 site 1020. In this option, the MAC addresses of remoteL2GWs 1022 that correspond to the Layer 2 site 1020, where a target hostresides, may be returned in response to local host's ARP/ND requests,e.g., when a host intends to communicate with an host in an remote Layer2 site 1020 and requests the address of the external host. Option 2 mayhave some advantages over option 1 in some situations.

According to the domain address restriction mechanism 1000, each L2GW1022 may be aware of all the hosts addresses in the same local Layer 2site 1020 of the L2GW 1022, e.g., using a reverse ARP scheme or othermethods. Each L2GW 1022 may also inform other L2GWs 1022 in other Layer2 sites 1020 of the hosts IP addresses and the corresponding VLANs (orVIDs).

To resolve addresses within one Layer 2 across the different sites, anARP/ND request may be sent from a first host 1026 (host A) to acorresponding local L2GW 1022 in a first site (Site 1). The host A maysend the ARP/ND request to obtain the MAC address of a second host 1026(host B) in a second site (Site 2). If the local L2GW 1022 has an entryfor the host B for the VLAN, e.g., the IP address of the host B underthe same VLAN, the local L2GW 1022 may respond to the ARP request bysending its own MAC address (option 1) or the MAC address of a secondL2GW 1022 associated with host B in Site 2 (option 2) to the host A. TheARP/ND request sent from one site, e.g., Site 1, may be intercepted bylocal L2GW 1022, and may not be forwarded (by the local L2GW 1022) toanother site. If the local L2GW 1022 does not comprise an entry for hostB under the same VLAN, the local L2GW 1022 may assume that host B doesnot exist and may not send a response to host A. The L2GWs 1022 of eachsite may send updates of their local hosts' IP addresses and theircorresponding VLAN on a regular or periodic basis to their peer L2GWs1022. It is possible that some L2GWs 1022 may not have received the IPaddresses of newly configured hosts in other locations. Typically, hostA may send ARP/ND request repetitively if no response is received.

FIG. 11 illustrates an embodiment of a data frame forwarding scheme 1100that may be used to forward messages or frames within one pseudo Layer 2network over multiple sites. The pseudo Layer 2 network over multiplesites may comprise a service provider or core network 1110 and aplurality of Layer 2 network domains 1120 that may be connected by theservice provider or core network 1110 via a plurality of edge nodes1112. The Layer 2 network domains 1120 may be located at one or more DCsites or locations and may comprise a plurality of L2GWs 1122 connectedto corresponding edge nodes 1112, and a plurality of intermediateswitches 1124 connected to corresponding L2GWs 1122. The intermediateswitches 1124 may also be connected to a plurality of hosts/servers/VMs1126. The components of the pseudo Layer 2 networks may be arranged asshown in FIG. 11 and may be similar to the corresponding components ofthe virtual Layer 2 network 200.

Based on the data frame forwarding scheme 1100, a first L2GW 1022 (GW1)may receive a first frame 1140, e.g., an Ethernet frame, from a firsthost 1126 (host A) in a first address domain 1120 (domain 1). The firstframe 1140 may be intended for a second host 1126 (host B) in a secondaddress domain 1120 (domain 2). The first frame 1140 may comprise aMAC-DA 1142 for a L2GW 1122 (GW). Host A may obtain the MAC address ofGW in an ARP response from GW1 in return to an ARP request for host B.GW may correspond to GW1 in domain 1 (according to option 1) or to asecond L2GW 1122 (GW2) in domain 2 (according to option 2). The firstframe 1140 may also comprise a MAC-SA 1144 for host A (A's MAC), anIP-DA 1146 for host B (B), an IP-SA 1148 for host A (A), and payload.

Based on option 1, GW1 may receive the first frame 1140, look up theVID/destination IP address of host B (e.g., as indicated by IP-DA 1146for host B), and replace the MAC-DA 1142 for GW in the first frame 1140with a MAC-DA 1162 for GW2 in an inner frame 1160. GW1 may also replacethe MAC-SA 1144 for host A (A's MAC) in the first frame 1140 with aMAC-SA 1164 for GW1 in the inner frame 1160. The inner frame 1160 mayalso comprise an IP-DA 1166 for host B (B), an IP-SA 1168 for host A(A), and payload. GW1 may send the inner frame 1160 to domain 2 via theservice provider or core network 1110. Based on option 2, GW1 may filterout all data frames intended for GW2 or any other external L2GW 1122,for instance based on an access list, replace the source addresses ofthe data frames (MAC-SA 1144 for host A or A's MAC) with GW1's own MACaddress, and then forward the data frames based on the destination MAC.

GW2 may receive the inner frame 1160 and process the inner frame 1160 totranslate the MAC addresses of the frame. Based on option 1, GW2 mayreceive the inner frame 1160, look up the VID/destination IP address ofhost B (e.g., as indicated by IP-DA 1166 for host B), and replace theMAC-DA 1162 for GW2 in the inner frame 1160 with a MAC-DA 1182 for hostB (B's MAC) in a second frame 1180. GW2 may also replace the MAC-SA 1164for GW1 in the inner frame 1160 with a MAC-SA 1184 for GW2 in the secondframe 1180. The second frame 1180 may also comprise an IP-DA 1186 forhost B (B), an IP-SA 1188 for host A (A), and payload. GW2 may then sendthe second frame 1180 to the destination host B. Based on option 2, GW2may only look up the VID/destination IP address of host B (e.g., asindicated by IP-DA 1166 for host B), and replace the MAC-DA 1162 for GW2with a MAC-DA 1182 for host B (B's MAC) in the second frame 1180.However, GW2 may keep the MAC-SA 1164.

As described above, GW2 may perform MAC address translation using theIP-DA 1166 for host B in the inner frame 1160 to find a correspondingMAC-DA 1182 for host B (B's MAC) in a second frame 1180. This MACtranslation step may require about the same amount of work as a NATscheme, e.g., for translating public IP address to private IP address.The MAC address translation in the data frame forwarding scheme 1100 maybe based on using the host IP address to find the corresponding MACaddress, while the NAT scheme is based on a TCP session.

FIG. 12 illustrates an embodiment of another data frame forwardingscheme 1200 that may be used to forward messages or frames betweenpseudo Layer 2 networks over multiple address domains. Specifically, thepseudo Layer 2 networks may be interconnected via an IP/MPLS network.The pseudo Layer 2 networks over the address domains may comprise anIP/MPLS network 1210 and a plurality of Layer 2 network domains 1220that may be connected to the IP/MPLS network 1210 via a plurality ofedge nodes 1212. The IP/MPLS network 210 may provide an IP service tosupport an inter domain between the address domains, e.g., the Layer 2network domains 1220. The Layer 2 network domains 1220 may be located atone or more DC sites or locations and may comprise a plurality of L2GWs1222 connected to corresponding edge nodes 1212, and a plurality ofintermediate switches 1224 connected to corresponding L2GWs 1222. Theintermediate switches 1224 may also be connected to a plurality ofhosts/servers/VMs 1226. The components of the pseudo Layer 2 networksmay be arranged as shown in FIG. 12 and may be similar to thecorresponding components of the virtual Layer 2 network 200.

Based on the data frame forwarding scheme 1200, a first L2GW 1022 (GW1)may receive a first frame 1240, e.g., an Ethernet frame, from a firsthost 1226 (host A) in a first address domain (domain 1). The first frame1240 may be intended for a second host 1226 (host B) in a second addressdomain (domain 2). The first frame 1240 may comprise a MAC-DA 1242 for aL2GW 1222 (GW). Host A may obtain the MAC address of GW in an ARPresponse from GW1 in return to an ARP request for host B. GW maycorrespond to GW1 in domain 1 (according to option 1) or to a secondL2GW 1222 (or GW2) in domain 2 (according to option 2). The first frame1240 may also comprise a MAC-SA 1244 for host A (A's MAC), an IP-DA 1246for host B (B), an IP-SA 1248 for host A (A), and payload.

GW1 may receive the first frame 1240 and process the frame based on oneof two options. In a first option, GW1 may receive the first frame 1240and add an IP header to obtain an inner frame 1250. The IP header maycomprise an IP-DA 1251 for GW2 and an IP-SA 1252 for GW1. GW1 may alsoprocess the first frame 1240 similar to the data frame forwarding scheme1100 to obtain in the inner frame 1250 a MAC-DA 1253 for GW2, a MAC-SA1254 for GW1, an IP-DA 1256 for host B (B), and an IP-SA 1257 for host(A). GW1 may send the inner frame 1250 to GW2 via the IP/MPLS network1210. GW2 may receive the inner frame 1250 and process the inner frame1250 similar to the data frame forwarding scheme 1100 to obtain a secondframe 1280 that comprises a MAC-DA 1282 for host B (B's MAC), a MAC-SA1284 for GW1 (according to option 1) or GW2 (according to options 2), anIP-DA 1286 for host B (B), an IP-SA 1288 for host A (A), and payload.GW2 may then forward the second frame 1280 to host B.

In a second option, GW1 may receive the first frame 1240 and replace theMAC-DA 1242 for GW in the first frame 1240 with an IP-DA 1262 for GW2 inan inner frame 1260. GW1 may also replace the MAC-SA 1244 for host A(A's MAC) in the first frame 1240 with an IP-SA 1264 for GW1 in theinner frame 1260. The inner frame 1260 may also comprise an IP-DA 1266for host B (B), an IP-SA 1268 for host A (A), and payload. GW1 may sendthe inner frame 1260 to GW2 via the IP/MPLS network 1210. GW2 mayreceive the inner frame 1260 and replace the IP-DA 1162 for GW2 in theinner frame 1260 with a MAC-DA 1282 for host B (B's MAC) in a secondframe 1280. GW2 may also replace the IP-SA 1264 for GW1 in the innerframe 1260 with a MAC-SA 1284 for GW2 (according to option 1) or GW1(according to options 2) in the second frame 1280. The second frame 1280may also comprise an IP-DA 1286 for host B (B), an IP-SA 1288 for host A(A), and payload. GW2 may then forward the second frame 1280 to host B.

In the above pseudo Layer 2 extension or networks across multipledomains, each L2GW may be configured for IP-MAC mapping of all the hostsin each VLAN in the L2GW's corresponding address domain. Each L2GW mayalso send IP addresses of all the hosts in each VLAN in thecorresponding address domain to other L2GWs in other address domains ona regular or periodic basis. Thus, the L2GWs in the address domains mayobtain IP addresses of hosts under each VLAN for all the address domainsof the pseudo Layer 2 network. The MAC addresses of the hosts in eachaddress domain may not be sent by the local L2GW to the L2GWs of theother address domains, which may substantially reduce the size of dataexchanged between the L2GWs. However, the L2GWs of different addressdomains may exchange among them the MAC addresses corresponding tonon-IP applications, e.g., if the number of non-IP applications isrelatively small. A BGP or similar method may be used to exchange theaddress information, including updates, between the L2GWs across theaddress domains.

Table 2 illustrates an example of mapping host addresses to thecorresponding L2GW's MAC addresses in pseudo Layer 2 networks. Aplurality of L2GW MAC addresses (e.g., GW-A MAC and GW-B MAC) may bemapped to a plurality of corresponding host addresses. Each L2GW MACaddress may be mapped to a plurality of host IP (or MAC) addresses in aplurality of VLANs (e.g., VID-1, VID-2, VID-n, . . . ), which may be inthe same address domain.

TABLE 2 IP-MAC Mapping L2GW VLAN Host GW-A MAC VID-1 IP addresses of allhosts in this VLAN (IP Prefix) MAC addresses (non-IP applications) VID-2IP addresses of all hosts in this VLAN (IP Prefix) MAC addresses (non-IPapplications) VID-n IP addresses of all hosts in this VLAN (IP Prefix)MAC addresses (non-IP applications) GW-B MAC

The pseudo Layer 2 extension or networks schemes above may restrict theMAC addresses of an address domain from being learned by anyswitches/servers/VMs in another address domain. The schemes may alsoprovide a scalable mechanism to connect substantially large Layer 2networks in multiple locations. In relatively large Layer 2 networksthat span across multiple address domains, the schemes may limit thenumber of MAC addresses that may be learned by any switch in the pseudoLayer 2 networks, where each switch may only learn the MAC addresses ofthe local address domain of the switch. The scheme may also providereachability discovery across multiple address domains using scalableaddress resolution across the address domains. Additionally, the schemesmay facilitate forwarding between address domains and the broadcast forunknown addresses, and support multicast groups.

FIG. 13 illustrates an embodiment of another data frame forwardingscheme 1300 that may be used to forward messages or frames betweenpseudo Layer 2 networks over multiple address domains and locations. Thedata frame forwarding scheme 1300 may be based on option 1 describedabove and may be used to forward frames from a host that moves from aprevious location to a new location in the pseudo Layer 2 networks andmaintains the same learned MAC address for a second host. The pseudoLayer 2 networks may comprise a service provider or core network 1310and a plurality of Layer 2 network domains 1320 that may be connected tothe service provider or core network 1310 via a plurality of edge nodes1312. The Layer 2 network domains 1320 may be located at multiple DCsites or locations and may comprise a plurality of L2GWs 1322 connectedto corresponding edge nodes 1312, and a plurality of intermediateswitches 1324 connected to corresponding L2GWs 1322. The intermediateswitches 1324 may also be connected to a plurality of hosts/servers/VMs1326. The components of the pseudo Layer 2 networks may be arranged asshown in FIG. 13 and may be similar to the corresponding components ofthe virtual Layer 2 network 200.

Based on the data frame forwarding scheme 1300, GW3 may receive a firstframe 1340, e.g., an Ethernet frame, from a first host 1326 (host A)after moving from Loc 1 to Loc 3. The frame 1340 may be intended for asecond host 1326 (host B) in Loc 2. The first frame 1340 may comprise aMAC-DA 1342 for GW1 in Loc 1, a MAC-SA 1344 for host A (A's MAC), anIP-DA 1346 for host B (B), an IP-SA 1348 for host A (A), and payload.GW3 may process the first frame 1340 and replace the MAC-SA 1344 forhost A (A's MAC) in the first frame 1340 with a MAC-SA 1354 for GW3 in afirst inner frame 1350, e.g., similar to the data frame forwardingscheme 1100. The first inner frame 1350 may also comprise a MAC-DA 1352for GW1, an IP-DA 1356 for host B (B), an IP-SA 1358 for host A (A), andpayload. GW3 may send the first inner frame 1350 to Loc 1 via theservice provider or core network 1310.

GW1 may receive the first inner frame 1350, look up the VID/destinationIP address of host B (e.g., as indicated by IP-DA 1356 for host B), andreplace the MAC-DA 1352 for GW1 in the first frame 1340 with a MAC-DA1362 for GW2 in a second inner frame 1360. The second inner frame 1360may also comprise a MAC-SA 1364 for GW3, an IP-DA 1366 for host B (B),an IP-SA 1368 for host A (A), and payload. GW1 may send the second innerframe 1360 to Loc 2 via the service provider or core network 1310.

GW2 may receive the second inner frame 1360 and process the second innerframe 1360 to translate the MAC addresses of the frame. GW2 may receivethe second inner frame 1360, look up the VID/destination IP address ofhost B (e.g., as indicated by IP-DA 1366 for host B), and replace theMAC-DA 1362 for GW2 in the inner frame 1360 with a MAC-DA 1382 for hostB (B's MAC) in a second frame 1380. GW2 may also replace the MAC-SA 1364for GW3 in the second inner frame 1360 with a MAC-SA 1384 for GW2. GW2may then send the second frame 1380 to the destination host B.

Further, host B may move from Loc 2 to another location, e.g., Loc 4(not shown). If GW2 has learned that host B has moved from Loc 2 to Loc4, then GW2 may send updates to its peers (other L2GWs 1322 in otherlocations). When a L2GW 1322 in Loc 4 (GW4) learns that host B is addedto its domain, GW4 may also update its peers. As such, each L2GW 1322may have updated address information about host B. If a L2GW 1322 hasnot learned that host B has moved from Loc 2 to Loc 4, then the L2GW1322 may still send a frame intended for host B from local hosts to Loc2. In turn, GW2 may receive and forward the frame in Loc 2, where theframe is lost since host B has moved from Loc 2. The frame may be losttemporarily until the frame is resent by the L2GW 1322 after host Bannounces its new location to the L2GW 1322.

FIG. 14 illustrates an embodiment of another data frame forwardingscheme 1400 that may be used to forward messages or frames betweenpseudo Layer 2 networks over multiple sites or domains. The data frameforwarding scheme 1400 may be based on option 2 described above and maybe used to forward frames from a host that moves from a previouslocation to a new location in the pseudo Layer 2 networks and maintainsthe same learned MAC address for a second host. The pseudo Layer 2networks may comprise a service network 1410 and a plurality of Layer 2network domains 1420 that may be connected by the service network 1410via a plurality of edge nodes 1412. The Layer 2 network domains 1420 maybe located at multiple DC sites or locations and may comprise aplurality of L2GWs 1422 connected to corresponding edge nodes 1412, anda plurality of intermediate switches 1424 connected (directly orindirectly) to corresponding L2GWs 1422. The intermediate switches 1424may also be connected (directly or indirectly) to a plurality ofhosts/servers/VMs 1426. The components of the pseudo Layer 2 networksmay be arranged as shown in FIG. 14 and may be similar to thecorresponding components of the virtual Layer 2 network 200.

Based on the data frame forwarding scheme 1400, GW3 may receive a firstframe 1440, e.g., an Ethernet frame, from a first host 1426 (host A)after moving from Loc 1 to Loc 3. The frame 1440 may be intended for asecond host 1426 (host B) in Loc 2. The first frame 1340 may comprise aMAC-DA 1442 for GW2 in Loc 2, a MAC-SA 1444 for host A (A's MAC), anIP-DA 1446 for host B (B), an IP-SA 1448 for host A (A), and payload.GW3 may process the first frame 1440 and replace the MAC-SA 1444 forhost A (A's MAC) in the first frame 1440 with a MAC-SA 1464 for GW3 inan inner frame 1460, e.g., similar to the data frame forwarding scheme1100. The inner frame 1460 may also comprise a MAC-DA 1462 for GW2, anIP-DA 1466 for host B (B), an IP-SA 1468 for host A (A), and payload.GW3 may send the inner frame 1460 to Loc 2 via the service provider orcore network 1410.

GW2 may receive the inner frame 1460 and process the inner frame 1460 totranslate the MAC addresses of the frame. GW2 may receive the innerframe 1460, look up the VID/destination IP address of host B (e.g., asindicated by IP-DA 1466 for host B), and replace the MAC-DA 1462 for GW2in the inner frame 1460 with a MAC-DA 1482 for host B (B's MAC) in asecond frame 1480. The inner frame 1460 may also have a MAC-SA 1484 forGW3. GW2 may then send the second frame 1480 to the destination host B.

Further, host B may move from Loc 2 to another location, e.g., Loc 4(not shown). If GW2 has learned that host B has moved from Loc 2 to Loc4, then GW2 may send updates to its peers (other L2GWs 1322 in otherlocations). When a L2GW 1322 in Loc 4 (GW4) learns that host B is addedto its domain, GW4 may also update its peers. As such, each L2GW 1322may have updated address information about host B. If a L2GW 1322 hasnot learned that host B has moved from Loc 2 to Loc 4, then the L2GW1322 may still send a frame intended for host B from local hosts to Loc2. In turn, GW2 may receive and forward the frame in Loc 2, where theframe is lost since host B has moved from Loc 2. The frame may be losttemporarily until the frame is resent by the L2GW 1322 after host Bannounces its new location to the L2GW 1322.

The pseudo Layer 2 extension or networks described above may supportaddress resolution in each address domain and may use a mechanism tokeep the L2GWs currently updated with IP addresses of all the hosts intheir domains/locations. Address resolution and IP address updating maybe implemented in one of two scenarios. The first scenario correspondsto when a host or VM is configured to send gratuitous ARP messages uponbeing added or after moving to a network. The second scenariocorresponds to when a host or VM that is added to or has moved to anetwork does not send ARP announcements. The two scenarios may behandled as described in the virtual Layer 2 networks above.

The virtual Layer 2 networks and similarly the pseudo Layer 2 networksdescribed above may support address resolution in each location/domainand a mechanism to keep each L2GW currently updated with IP addresses ofits local hosts in its location/domain. In one scenario, when a host ora VM is added to the network, the host or VM may send an ARPannouncement, such as a gratuitous ARP message, to its Layer 2 networkor local area. In another scenario, the host or VM added to the networkmay not send an ARP announcement.

In the first scenario, a new VM in a Layer 2 network or location/domainmay send a gratuitous ARP message to a L2GW. When the L2GW receives thegratuitous ARP message, the L2GW may update its local IPAddrTable butmay not forward the gratuitous ARP message to other locations/domains orLayer 2 networks. Additionally, the L2GW may use a timer for each entryin the IPAddrTable to handle the case of shutting down or removing ahost from a location/domain. If the timer of an entry is about toexpire, the L2GW may send an ARP (e.g., via uni-cast) to the host of theentry. Sending the ARP as a uni-cast message instead of broadcasting theARP may avoid flooding the local Layer 2 domain of the host and theL2GW. When a host moves from a first location to a second location, aL2GW may receive an update message from the first location and/or thesecond location. If the L2GW detects that the host exists in both thefirst location and the second location, the L2GW may send a local ARPmessage in the first location to verify that the host does not existanymore in the first location. Upon determining that the host is nolonger present in the first location, for example if no response to theARP message is detected, the L2GW may update its local IPAddrTableaccordingly. If the L2GW receives a response for the ARP message for itsown location, then a MAC multi-homing mechanism of BGP may be used.

In the second scenario, the new host in a location may not send an ARPannouncement. In this case, when an application (e.g., at a host) needsto resolve the MAC address for an IP host, the application may send outan ARP request that may be broadcasted in the location. The ARP requestmay be intercepted by a L2GW (or a Top-of-Rack (ToR) switch), e.g., byimplementing a proxy ARP function. In a relatively large DC, the L2GWmay not be able to process all the ARP requests. Instead, a plurality ofL2GW delegates (e.g., ToR switches) may intercept the ARP announcements.The L2GW may push down the IP addresses (e.g., a summary of IPaddresses) that are learned from other locations to its correspondingdelegates (ToR switches). The delegates may then intercept the ARPrequests from hosts or local servers. If an IP address in the ARPrequest from a host or server is present in the IPAddrTable of the L2GW,the L2GW may return an ARP response with the L2GW's MAC address to thehost or server, without forwarding the broadcasted ARP request anyfurther. For non-IP applications, e.g., applications that run directlyover Ethernet without IP, the applications may use MAC addresses as DAswhen sending data. The non-IP applications may not send an ARP messageprior to sending the data frames. The data frames may be forwarded usingunknown flooding or Multiple MAC registration Protocol (MMRP).

In one scenario, an application (e.g., on a host) may send a gratuitousARP message upon joining one of the interconnected Layer 2 networks inone location to obtain a MAC address for a targeted IP address. The L2GWor its delegate (e.g., ToR switch) may receive the ARP request and checkits IP host table. If the IP address is found in the table, the L2GW maysend an ARP reply to the application. The L2GW may send its MAC addressin the reply if the targeted IP address corresponds to an IP host inanother location. If the IP address is not found, no reply may be sentfrom the L2GW, which may maintain the current or last updated IPaddresses of the hosts in all locations. In relatively large DCs,multiple L2GWs may be used, e.g., in the same location, where each L2GWmay handle a subset of VLANs. As such, each L2GW may need to maintain asubset of IP addresses that comprise the IP addresses of the hosts inthe corresponding VLAN.

In the case of substantially large DCs, e.g., that comprise tens ofthousands of VMs, it may be difficult for a single node to handle allthe ARP requests and/or gratuitous ARP messages. In this case, severalschemes may be considered. For instance, a plurality of nodes or L2GWsmay be used to handle different subsets of VLANs within a DC, asdescribed above. Additionally or alternatively, multiple delegates maybe assigned for a L2GW in each location. For instance, a plurality ofToR switches or access switches may be used. Each L2GW's delegate may beresponsible for intercepting gratuitous ARP messages on itscorresponding downlinks or in the form of a Port Binding Protocol. Thedelegates may send a consolidated address list (AddressList) to theirL2GWs. The L2GW may also push down its learned IP address lists fromother locations to its delegates. If there are multiple L2GWs in alocation that are responsible for different subsets of VLANS, thedelegates may need to send a plurality of consolidated messages thatcomprise each the AddressLists in the VLANs associated with thecorresponding L2GWs.

In comparison to Cisco's OTV scheme, using the virtual Layer 2 networkdescribed above may substantially reduce the size of forwarding tableson intermediate switches in each location. The switches in one locationmay not need to learn MAC addresses of IP hosts in other locations,e.g., assuming that the majority of hosts run IP applications. Thisscheme may also substantially reduce the size of the address informationexchanged among the L2GWs. For example, a subnet that may comprisethousands of VMs may be mapped to a L2GW MAC address. The hierarchicalLayer 2 scheme of the virtual Layer 2 network may use 802.1ah standard,which may be supported by commercial Ethernet chip sets, while Cisco'sscheme uses proprietary IP encapsulation. Both schemes may use peerlocation gateway device (L2GW) address as outer destination address. Thehierarchical Layer 2 scheme may also use address translation, which maybe supported by current IP gateways. However, the hierarchical Layer 2scheme may use MAC address translation instead of IP addresstranslation. The MAC address translation may need carrier grade NATimplementation that can perform address translation for tens ofthousands of addresses.

In an embodiment, a VLAN may span across multiple locations. Thus, amulticast group may also span across multiple locations. Specifically,the multicast group may span across a subset of locations in the virtualLayer 2 network. For example, if there are about ten locations in thevirtual Layer 2 network, the multicast group may only span across threeof the ten locations. A multicast group within one service instance maybe configured by a network administrator system (NMS) or may beautomatically established in Layer 2 using MMRP. Since L2GW supports802.1ah, the L2GW may have a built-in component to map client multicastgroups to proper multicast groups in the core network. In a worst casescenario, the L2GW may replicate the multicast data frames to all thelocations of the service instance. For example, according to Microsoftresearch data, about one out of four traffic may go to a differentlocation. Thus, the replication by L2GW may be simpler than implementinga complicated mechanism in the Provider core.

The virtual Layer 2 network may support broadcast traffic, such as forARP requests and/or Dynamic Host Configuration Protocol (DHCP) requests.The broadcast traffic may be supported by creating multiple ARPdelegates, such as ToR switches, in each location. The broadcast trafficmay also be supported by adding a new component to the Port BindingProtocol for the delegates to maintain current updates of all the IPhosts from the servers. Additionally, the L2GW may push down on aperiodic or regular basis all the learned host IP addresses from otherlocations.

In some instances, the L2GW may receive unknown DAs. The L2GW may keepcurrent updates of all the hosts (or applications) in its location andperiodically or regularly push its address information to all the peers(other L2GWs in other locations). If the L2GW receives a framecomprising an unknown DA, the L2GW may broadcast the frame to the otherlocations. To avoid attacks on the network, a limit may be imposed onthe maximum number of times the L2GW may forward or broadcast a receivedunknown DA. The L2GW may be configured to learn the addresses of theintermediate switches in another location to avoid mistaking anintermediate switch address for an unknown address before sending theaddress to the other location. Although there may be tens of thousandsof VMs in each DC location, the number of switches in each DC may belimited, such as the number of ToR or access switches, end of row oraggregation switches, and/or core switches. The L2GW may learn the MACaddresses of all the intermediate switches in a location ahead of time,e.g., via a Bridge Protocol Data Unit (BPDU) from each switch. Messagesmay not be sent directly to the intermediate switches, except formanagement system or Operations, Administration, and Maintenance (OAM)messages. An intermediate switch that expects or is configured toreceive NMS/OAM messages may allow other switches in the location tolearn its MAC address by sending an autonomous message to NMS or a MMRPannouncement.

In some embodiments, the L2GWs may use BGP, e.g., instead of IS-IS, forexchanging address information. A plurality of options may be used forcontrolling Layer 2 (L2) communications. For instance, forwardingoptions may include Layer 2 only forwarding with MAC and MAC, Layer 2forwarding over MPLS, and Layer 2 forwarding in Layer 3 network. Optionsof Layer 2 control plane may include Layer 2 IS-IS mesh control, Layer2.5 MPLS static control, Label Distribution Protocol (LDP), ResourceReservation Protocol (RSVP)-Traffic Engineering (TE) using InteriorGateway Protocol (IGP) Constraint-based Shortest Path First (CSFP), andBGP discovery. Some VLAN mapping issues may also be considered, such asthe VLAN-MAC mapping required for uniqueness and whether Network BridgedVLANs (e.g., VLANs-4K) may be too small for a DC. Table 3 illustrates aplurality of control plane options that may be used for Layer 2 controlplane. The options may be based on IEEE 802.1ah, IEEE 802.1q, and IEEE802.1aq, all of which are incorporated herein by reference. Table 4illustrates some of the advantages and disadvantages (pros and cons) ofthe control plane options in Table 2.

TABLE 3 Layer 2 Control Lane Options MPLS control Transport L2 controlplane plane IGP-OSPF/IS-IS BGP L2 Provider 802.1q Not Pass IP-MACInternal BGP Backbone Bridge 802.1ah applicable mapping (IBGP) mesh(PBB) External BGP (EBGP) mesh VPLS (MPLS) MAC learning LDP for IGP forCSPF BGP auto- interaction domain discovery of end with L2 RSVP-TEpoints MPLS static VPLS ARP Mediation L2 over IP L2 only with DC NotPeer validation Peer validation (802.1aq) applicable Peer connectivityPeer path Pass IP-MAC connectivity mapping IP-Mapping Explicitdistribution multithreading (XMT)

TABLE 4 Control plane options IGP-Open Shortest L2 control MPLS controlPath First Transport plane plane (OSPF)/IS-IS BGP L2 PBB No Layer 3 VPLSis done Pros: Pros: configuration IS-IS pass MAC BGP policy address BGPauto-discovery used Multithread (MT)- for the L2 PBB to VPLS VPN mapping−>VLAN BGP efficient for Cons: efficiency for large number of peers andIP mapping I-MAC mappings Multiple VLANs VPLS MAC Pros: Done Pros: Pros:Same as above (MPLS) learning Cons: CSPF for IS- Cons: interaction CodeIS/OSPF BGP inter-domain with L2 overhead, Fast peer MPLS interactionwith multicast not convergence MPLS Layer 3 (L3) VPN efficient MTtopology Cons: not efficient with A) large number of peers B) largenumber of IP-MAC mappings L2 over IP Limited to Not applicable Peervalidation Peer validation only DC Peer connectivity Peer pathconnectivity IP to MAC mapping IP-Mapping distribution XMT

There may be a plurality of differences between Cisco's OTV and the BGPthat may be supported in the virtual Layer 2 network. For instance, OTVbasic aspects may include OTV multicast groups, OTV IS-IS usage, whichmay require MT-IS-IS, and OTV forwarding. Additionally, BGP may supportBGP-MAC mapping and IP overlay, such as for DC multicast group. BGP-MACmapping may also use MT-BGP. Further, IBGP may be supported by MT-IS-ISand using IS-IS for peer topology (e.g., Label Switched PathVerification (LSVP)).

In the virtual Layer 2 network above, the number of applications withinone Layer 2 network (or DC) may increase substantially, e.g., over time.Thus, a mechanism may be needed to avoid issues associated withsubstantially large Layer 2 networks. These issues may includeunpredictable behavior of servers/hosts and their applications. Forexample, the servers/hosts may correspond to different vendors, wheresome may be configured to send ARP messages and others may be configuredto broadcast messages. Further, typical lower cost Layer 2 switches maynot have sophisticated features to block broadcast data frames to havepolicy implemented to limit flooding and broadcast. Hosts orapplications may also age out MAC addresses to target IP mappingfrequently, e.g., in about minutes. A host may also frequently send outgratuitous ARP messages, such as when the host performs a switch over(from active to standby) or when the host has a software glitch. In somecases, the Layer 2 network components are divided into smaller subgroupsto confine broadcast into a smaller number of nodes.

FIG. 15 illustrates an embodiment of a typical broadcast scheme 1500that may be used in a Layer 2 network/domain, e.g., a VLAN, which may bepart of the virtual Layer 2 networks or the pseudo Layer 2 networksabove. The Layer 2 network/domain or VLAN may comprise a plurality ofaccess switches (ASs) 1522 located in a Pod 1530, e.g., in a DC. TheVLAN may also comprise a plurality of closed user groups (CUGs) 1535connected to the ASs 1522. Each CUG 1535 may comprise a plurality ofEnd-of-Row (EoR) switches 1524 connected to the ASs 1522, a plurality ofToR switches 1537 connected to the EoR switches 1524, and a plurality ofservers/VMs 1539 connected to the ToR switches 1537. The ASs 1522 may beconnected to a plurality of Pods (not shown) in other DCs that maycorrespond to other Layer 2 networks/domains of the virtual Layer 2networks or the pseudo Layer 2 networks. The components of the Layer 2network/domain or the Pod 1530 may be arranged as shown in FIG. 15.

The typical broadcast scheme 1500 may suffer from broadcast scalabilityissues. For instance, frames with unknown DAs may be flooded within thePod 1530 to all the end systems in the VLAN. For example, the frameswith unknown DAs may be flooded to all or a plurality of servers/VMs1539 in the ASs 1522 in the CUGs 1535, as indicated by the dashed arrowsin FIG. 15. The frames with unknown addresses may also be flooded in theopposite direction, via an AS 1522, to a plurality of other Pods (inother DCs) in the core, which may be associated with the same service asthe Pod 1530. The frames may be further flooded to a plurality of VMs inthe other Pods, which may reach thousands of VMs. Such broadcast schemefor unknown DAs may not be efficient in relatively large networks, e.g.,that comprise many DCs.

FIG. 16 illustrates an embodiment of another broadcast scheme 1600 thatmay be used in a Layer 2 network/domain, e.g., a VLAN, which may be partof the virtual Layer 2 networks or the pseudo Layer 2 networks above.The broadcast scheme 1600 may be more controlled and thus more scalableand efficient than the broadcast scheme 1500. The Layer 2 network/domainor VLAN may comprise a plurality of ASs 1622 located in a Pod 1630,e.g., in a DC. The VLAN may also comprise a plurality of CUGs 1635connected to the ASs 1622. Each CUG 1635 may comprise a plurality of EoRswitches 1624 connected to the ASs 1622, a plurality of ToR switches1637 connected to the EoR switches 1624, and a plurality of servers/VMs1639 connected to the ToR switches 1637. The ASs 1622 may be connectedto a plurality of Pods (not shown) in other DCs that may correspond toother Layer 2 networks/domains of the virtual Layer 2 networks or thepseudo Layer 2 networks. The components of the Layer 2 network/domain orthe Pod 1630 may be arranged as shown in FIG. 16.

To control or limit the broadcast scope of the broadcast scheme 1600,frames with unknown DAs may only be flooded within the Pod 1530 to asingle root, for instance to one server/VM 1639 that may be designatedas a broadcast server or to an AS 1622. The frames may be flooded to theroot using a rooted-multipoint (RMP) VLAN configuration, e.g., a pushVLAN tag for RMP VLAN that is rooted at a broadcast server. However, theflooded frame may not be forwarded to all the other servers, e.g., thatare not broadcast servers, which may save link resources and serverprocessing of extraneous frames. Additionally, the forwarded frames maynot be forwarded to the core, e.g., to other Pods or DCs.

In some embodiments, the broadcast server may host a proxy ARP server, aDHCP server, and/or other specific function servers, e.g., for improvingefficiency, scalability, and/or security. For instance, the broadcastserver may be configured to provide security in DCs that only allowselected broadcast services. If no known service is selected, dataframes with unknown DAs may be flooded from the broadcasts server on afirst or original VLAN. The broadcast scheme 1600 may be used to handlecases where customer applications are allowed to use Layer 2 broadcast.A data rate limiter may also be used to protect against broadcaststorms, e.g., avoid substantial broadcast traffic.

As described above, when introducing server virtualization in DCs, thenumber of hosts in a DC may increase substantially, e.g., over time.Using server virtualization, each physical server, which may originallyhost an end-station, may become capable of hosting hundreds ofend-stations or VMs. The VMs may be added, deleted, and/or movedflexibly between servers, which may improve performance and utilizationof the servers. This capability may be used as a building block forcloud computing services, e.g., to offer client controlled virtualsubnets and virtual hosts. The client control virtual subnets offered bycloud computing services may allow clients to define their own subnetswith corresponding IP addresses and policies.

The rapid growth of virtual hosts may substantially impact networks andservers. For instance, one resulting issue may be handling frequent ARPrequests, such as ARP IP version 4 (IPv4) requests, or neighbordiscovery (ND) requests, such as ND IP version 6 (IPv6) requests fromhosts. The hosts in a DC may send out such requests frequently duecaches or entries that may age in about few minutes. In the case of tensof thousands of hosts in a DC, which may have different MAC addresses,the amount of ARP or ND messages or requests per second may reach morethan about 1,000 to 10,000 requests per second. This rate or frequencyof requests may impose substantial computational burden on the hosts.Another issue associated with a substantially large number of virtualhosts in a DC may be existing duplicated IP addresses within one VLAN,which may affect the ARP or ND scheme from working properly. Some loadbalancing techniques may also require multiple hosts which serve thesame application to use the same IP address but with different MACaddresses. Some cloud computing services may allow users to use theirown subnets with IP addresses and self defined policies among thesubnets. As such, it may not be possible to designate a VLAN per eachclient since the maximum number of available VLANS may be about 4095 insome systems while there may be hundreds of thousands of client subnets.In this scenario, there may be duplicated IP addresses in differentclient subnets that end up in one VLAN.

In an embodiment, a scalable address resolution mechanism that may beused in substantially large Layer 2 networks, which may comprise asingle VLAN that includes a substantial number of hosts, such as VMsand/or end-stations. Additionally, a mechanism is described for properaddress resolution in a VLAN with duplicated IP addresses. The mechanismmay be used for both ARP IPv4 addresses and ND IPv6 addresses.

FIG. 17 illustrates an embodiment of interconnected network districts1700 in a bridged Layer 2 network, e.g., an Ethernet. The bridged Layer2 network may comprise a plurality of core bridges 1712 in a coredistrict 1710, which may be connected to a plurality of districts 1720.The Layer 2 bridged network may also comprise a plurality of DBBs 1722that may be part of the core district 1710 and the districts 1720, andthus may interconnect the core district 1710 and the districts 1720.Each district 1720 may also comprise a plurality of intermediateswitches 1724 connected to corresponding DBBs 1722, and a plurality ofend-stations 1726, e.g., servers/VMs, connected to correspondingintermediate switches 1724. The components of the interconnected networkdistricts 1700 may be arranged as shown in FIG. 17.

FIG. 18 illustrates another embodiment of interconnected networkdistricts 1800 that may be configured similar to the interconnectednetwork districts 1700. The interconnected network districts 1800 maycomprise a plurality of core bridges 1812 and a plurality of DBBs 1822(e.g., ToR switches) or district boundary switches in a core district1810. The interconnected network districts 1800 may also comprise aplurality of intermediate switches 1824 and a plurality of end-stations1826, e.g., servers/VMs, in a plurality of districts 1820. The districts1820 may also comprise the DBBs 1822 that connected the districts 1820to the core district 1810. The components of the interconnected networkdistricts 1800 may be arranged as shown in FIG. 18. A VLAN may beestablished in the interconnected network districts 1800, as indicatedby the bold solid lines in FIG. 18. The VLAN may be associated with aVID and may be established between one of the core bridges 1812 in thecore bridge 1810, a subset of the DBBs 1822 in the districts 1820, and asubset of intermediate switches 1824 and servers/VMs 1826 in thedistricts 1820.

The DBBs 1822 in districts 1820 may be aware and maintain a <MAC,VID>pair for each end-station 1826 in the districts 1820. This addressinformation may be communicated by the end-stations 1826 to thecorresponding DBBs 1822 in the corresponding districts 1820 via EdgeVirtual Bridging (EVB) Virtual Station Interface (VSI) Discovery andConfiguration Protocol (VDP). The DBB 1822 may also register thisinformation with the other DBBs 1822, e.g., via MMRP. Alternatively, theaddress information may be communicated by the end-stations 1826 totheir DBBs 1822 using gratuitous ARP messages or by sendingconfiguration messages from a NMS.

In an embodiment, a scalable address resolution mechanism may beimplemented to support a VLAN that comprise a relatively large number ofhosts in the interconnected network districts 1800. Specifically, theMAC address of a DBB 1822 in one district 1820 and the VID of the VLANmay be used as a response to an ARP request for the district's hostaddresses from other districts 1820. In some cases, a DS may beconfigured to obtain summarized address information for the end-stations1826 in a district 1820 when the DS may not be capable of handling arelatively large number of messages for individual end-stations 1826 orhosts. In such cases, the DBB 1822 in a district 1820 may terminate allgratuitous ARP messages for the districts hosts or snoop all thegratuitous ARP messages sent from its district 1820, and send outinstead a gratuitous group announcement, e.g., that summarizes the hostsaddress information for the DS. The DBB may send its own gratuitous ARPannouncement to announce all the host IP addresses in its district 1820to other districts 1820.

Further, the DBB 1822 in a district 1820 may serve as an ARP proxy bysending its own MAC address to other districts 1820, e.g., via a corebridge 1812 in the core district 1810. The core bridges 1812 may only beaware of the MAC addresses of the DBBs 1822 in the districts 1820 butnot the MAC addresses of the intermediate switches 1824 and end-stations1826 or hosts, which makes this scheme more scalable. For instance, whena first end-station 1826 in a first district 1820 sends an ARP requestfor the address of a second end-station 1826 in a second district 1820,the MAC address of a DBB 1822 of the second district 1820 may bereturned in response to the first end-station 1826.

FIG. 19 illustrates an embodiment of ARP proxy scheme 1900 that may beused in a Layer 2 bridged network, e.g., for the interconnected networkdistricts 1800. The Layer 2 bridged network may comprise a core district1910, a plurality of DBBs 1922 or district boundary switches connectedto the core district 1910, and a plurality of end-stations 1926 (e.g.,VMs) connected to corresponding DBBs 1922 in their districts. The Layer2 bridged network may also comprise a DS 1940 that may be connected tothe DBBs 1922, e.g., via the core district 1910. The DBBs 1922 andend-stations 1926 may belong to a VLAN established in the Layer 2bridged network and associated with a VID. The components of the Layer 2bridged network may be arranged as shown in FIG. 19.

Based on the ARP proxy scheme 1900, a first DBB 1922 (DBB X) mayintercept an ARP request from a first end-station 1926 in its localdistrict. The ARP request may be for a MAC address for a secondend-station 1926 in another district. The ARP request may comprise theIP DA (10.1.0.2) of the second end-station 1926, and the IP sourceaddress (SA) (10.1.0.1) and MAC SA (A) of the first end-station 1926.The first end-station 1926 may maintain the IP addresses of the otherend-stations 1922 in a VM ARP table 1960. DBB X may send a DS query toobtain a MAC address for the second end-station 1926 from the DS 1940.The DS query may comprise the IP address (10.1.0.2) of the secondend-station 1926, and the IP SA (10.1.0.1) and MAC SA (A) of the firstend-station 1926. The DS 1940 may maintain the IP addresses, MACaddresses, and information about the associated DBBs 1922 or locationsof the end-stations 1926 (hosts) in a DS address table 1950.

The DS 1940 may then return to DBB X a DS response that comprises the IPaddress (10.1.0.2) of the second end-station 1926 and the MAC address(Y) of a second DBB 1926 (DBB Y) associated with the second end-station1926 in the other district, as indicated in the DS address table 1950.In turn, DBB X may send an ARP response to the first end-station 1926that comprises the IP DA (10.1.0.1) and MAC DA (A) of the firstend-station 1926, the IP SA (10.1.0.2) of the second end-station 1926,and the MAC address of DBB Y (Y). The first end-station 1926 may thenassociate the MAC address of DBB Y (Y) with the IP address (10.1.0.2) ofthe second end-station 1926 in the VM ARP table 1960. The firstend-station 1926 may use the MAC address of DBB Y as the DA to forwardframes that are intended for the second end-station 1926.

In the ARP proxy scheme 1900, the DBBs 1922 may only need to maintainthe MAC addresses of the other DBBs 1922 in the districts without theMAC and IP addresses of the hosts in the districts. Since the DAs in thedata frames sent to the DBBs 1922 only correspond to DBBs MAC addresses,as described above, the DBBs 1922 may not need to be aware of the otheraddresses, which makes this scheme more scalable.

FIG. 20 illustrates an embodiment of a data frame forwarding scheme 2000that may be used in a Layer 2 bridged network, e.g., for theinterconnected network districts 1800. The Layer 2 bridged network maycomprise a core district 2010, a plurality of DBBs 2022 or districtboundary switches in a plurality of districts 2020 connected to the coredistrict 2010, and a plurality of intermediate switches 2024 andend-stations 2026 (e.g., VMs) connected to corresponding DBBs 2022 intheir districts 2020. Some of the DBBs 2022, intermediate switches 2024,and end-stations 2026 across the districts 2020 may belong to a VLANestablished in the Layer 2 bridged network and associated with a VID.The components of the Layer 2 bridged network may be arranged as shownin FIG. 20.

The data frame forwarding scheme 2000 may be based on MAT at the DBBs2022, which may be similar to IP NAT. The MAT may comprise using innerIP DAs and ARP tables to find corresponding MAC DAs. For instance, afirst DBB 2022 (DBB1) may receive a frame 2040, e.g., an Ethernet frame,from a first end-station 2026 (host A) in a first district (district 1).The frame 2040 may be intended for a second end-station 2026 (host B) ina second district (district 2). The frame 2040 may comprise a MAC-DA2042 for a second DBB in district 2 (DBB2), a MAC-SA 2044 for host A(A's MAC), an IP-DA 2046 for host B (B), an IP-SA 2048 for host A (A),and payload. DBB 1 may forward the frame 2040 to district 2 via the coredistrict 2010. A second DBB 2022 (DBB2) in district 2 may receive theframe 2040 and replace the MAC-DA 2042 for DBB2 (DBB2) in the frame 2040with a MAC-DA 2082 for host B (B's MAC) in a second frame 2080. DBB2 maydetermine B's MAC based on the IP-DA 2046 for host B (B) and acorresponding entry in its ARP table. The second frame may also comprisea MAC-SA 2084 for host A (A's MAC), an IP-DA 2086 for host B (B), anIP-SA 2088 for host A (A), and payload. DBB2 may send the second frame2080 to host B in district 2. Since the SAs in the received frames atdistrict 2 are not changed, the data frame forwarding scheme 2000 maynot affect implemented DHCP in the network.

In the network above, the core bridges or switches of the core district,e.g., the core bridges 1812 in the core district 1810, may only need tomaintain the MAC addresses of the DBBs in the districts without the MACand IP addresses of the hosts in the districts. Since the DAs in thedata frames forwarded through the core district may only correspond toDBBs MAC addresses, as described above, the core bridges may not need tobe aware of the other addresses. The MAC addresses of the DBBs may bemaintained in the core bridges' forwarding databases (FDBs). The corebridges or switches may learn the topology of all the DBBs via a linkstate based protocol. For example, the DBBs may send out link stateadvertisements (LSAs), e.g., using IEEE 802.1aq, TransparentInterconnect of Lots of Links (TRILL), or IP based core. If SpanningTree Protocol (STP) is used among the core bridges, MAC address learningmay be disabled at the core bridges. In this case, the DBBs may registerthemselves with the core bridges.

In an embodiment, the DBBs may act as ARP proxies, as described above,if a DS is not used. Gratuitous ARP messages may be sent by theend-stations to announce their own MAC addresses. Gratuitous groupannouncements may also be sent by the DBBs to announce their own MACaddresses and the IP addresses for all the hosts within their localdistricts. The gratuitous group announcements may be used to announcethe MAC and IP addresses to the other DBBs in the other districts. Theannounced MAC addresses and IP addresses may be used in the other DBBSto translate DBB MAC DAs in received frames according to host IP DAs. Agratuitous group ARP may be sent by a DBB to announce a subset of hostIP addresses for each VLAN associated with the DBB. The gratuitous groupARP may comprise a mapping of subsets of host IP addresses to aplurality of VLANs for the DBB.

Table 5 illustrates an example of mapping host IP addresses to thecorresponding DBB MAC addresses in the interconnected districts. Themapping may be sent in a gratuitous group ARP by a DBB to announce itshost IP addresses for each VLAN associated with the DBB. A DBB MACaddress (DBB-MAC) may be mapped to a plurality of corresponding host IPaddresses. Each DBB MAC address may be mapped to a plurality of host IPaddresses in a plurality of VLANs (e.g., VID-1, VID-2, VID-n, . . . ),which may be in the same or different districts.

TABLE 5 Information carried by Gratuitous Group ARP DBB VLAN HostDBB-MAC VID-1 IP addresses of all hosts in this VLAN (IP Prefix) VID-2IP addresses of all hosts in this VLAN (IP Prefix) VID-n IP addresses ofall hosts in this VLAN (IP Prefix)

In some situations, multiple hosts in the interconnected districts mayhave the same IP addresses and may be associated with the same VLAN (orVID). For instance, a virtual subnet of a cloud computing service mayallow clients to name their own private IP addresses. The number ofvirtual subnets offered by a cloud computing service may substantiallyexceed the total number of allowed VLANs (e.g., about 4095 VLANs). Assuch, a plurality of virtual hosts (e.g., VM or virtual end-stations)may use or be allowed to have the same IP addresses but with differentMAC addresses. In other instances, multiple end-stations may serve thesame application using the same IP addresses but different MACaddresses.

In an embodiment, a DBB may be assigned a plurality of MAC addresses,referred to herein as delegate MAC addresses, e.g., to differentiatebetween different hosts that use the same (duplicated) IP address. TheDBB may also be associated with a plurality of VLANs. Further, each VLANon the DBB may be associated with a plurality of subnets or virtualsubnets, e.g., that comprise different subsets of hosts within the VLAN.The virtual subnets may be associated with a plurality of subnetidentifiers (IDs). If the number of duplicated IP addresses for thehosts is substantially less than the number of virtual subnets of theVLAN, then the number of delegate MAC addresses for the DBB may also besubstantially less.

FIG. 21 illustrates an embodiment of an ARP proxy scheme 2100 that maybe used for interconnected network districts in a Layer 2 bridgednetwork. The Layer 2 bridged network may comprise a core district 2110,a plurality of DBBs 2122 or district boundary switches connected to thecore district 2110, and a plurality of end-stations 2126 (e.g., VMs)connected to corresponding DBBs 2122 in their districts. The Layer 2bridged network may also comprise a DS 2140 that may be connected to theDBBs 2122, e.g., via the core district 2110. The DBBs 2122 andend-stations 2126 may belong to a VLAN established in the Layer 2bridged network. The components of the Layer 2 bridged network may bearranged as shown in FIG. 21.

Based on the ARP proxy scheme 2100, a first DBB 2122 (DBB X) mayintercept an ARP request from a first end-station 2126 in its localdistrict. The ARP request may be for a MAC address for a secondend-station 2126 in another district. The ARP request may comprise theIP DA (10.1.0.2) of the second end-station 2126, and the IP SA(10.1.0.1) and MAC SA (A) of the first end-station 2126. The firstend-station 2126 may maintain the IP addresses of the other end-stations2126 in a VM ARP table 2160. DBB X may then forward a DS query to obtaina MAC address for the second end-station 2126 from the DS 2140. The DSquery may comprise the IP address (10.1.0.2) of the second end-station2126, and the IP SA (10.1.0.1) and MAC SA (A) of the first end-station2126. The DS 2140 may maintain the IP addresses, MAC addresses, VLAN IDsor VIDs, customer (virtual subnet) IDs, and information about theassociated DBBs 2122 or locations of the end-stations 2126 in a DSaddress table 2150.

The DS 2140 may use the MAC SA (A) in the DS query to determine whichcustomer (virtual subnet) ID belongs to the requesting VM (firstend-station 2126). For example, according to the DS address table 2150,the customer ID, Joe, corresponds to the MAC SA (A). The DS 2140 maythen return to DBB X a DS response that comprises the IP address(10.1.0.2) of the second end-station 2126 and a delegate MAC address(Y1) of a second DBB 2122 (DBB Y) associated with the customer ID (Joe)of the first end-station 2126. In turn, DBB X may send an ARP responseto the first end-station 2126 that comprises the IP DA (10.1.0.1) andMAC DA (A) of the first end-station 2126, the IP SA (10.1.0.2) of thesecond end-station 2126, and the delegate MAC address of DBB Y (Y1). Thefirst end-station 2126 may then associate the delegate MAC address ofDBB Y (Y1) with the IP address (10.1.0.2) of the second end-station 2126in the VM ARP table 2160. The first end-station 2126 may use thedelegate MAC address of DBB Y as the DA to forward frames that areintended for the second end-station 2126.

A third end-station 2126 in another district may also send an ARPrequest (for the second end-station 2126 to a corresponding local DBB2122 (DBB Z) in the third end-station's district. DBB Z may thencommunicate with the DS 2140, as described above, and return accordinglyto the third end-station 2126 an ARP response that comprises the IP DA(10.1.0.3) and MAC DA of the third end-station 2126, the IP SA(10.1.0.2) of the second end-station 2126, and a delegate MAC address ofDBB Y (Y2) associated with the customer ID, Bob, of the thirdend-station 2126 in the DS address table 2150. The third end-station2126 may then associate the delegate MAC address of DBB Y (Y2) with theIP address (10.1.0.2) of the second end-station 2126 in a VM ARP table2170 of the third end-station 2126. The third end-station 2126 may usethis delegate MAC address of DBB Y as the DA to forward frames that areintended for the second end-station 2126.

Table 6 illustrates an example of mapping a duplicated host IP addressto corresponding delegate DBB MAC addresses in a VLAN in theinterconnected districts. The duplicated host address may be used by aplurality of hosts for one intended application or host. The delegateMAC DBB addresses may be assigned for the different hosts that use thesame application (or communicate with the same host). For each VLAN, ahost IP address may be mapped to a plurality of delegate DBB MACaddresses (MAC-12, MAC-13, MAC-14, . . . ) for a plurality of hosts,e.g., associated with different subnets of the VLAN. The delegate DBBMAC addresses may also be associated with a base (original) DBB MACaddress (MAC-11). The base and delegate DBB MAC addresses for the sameIP may be different for different VLANs. When a VLAN does not havedelegate addresses, the DBB base address may be used for the VLAN. Ifthere are about 10 duplicated IP addresses within one VLAN, then about10 columns (ten MAC addresses) in the table 6 may be used.

TABLE 6 MAT for Duplicated IP addresses. DBB Base DBB DBB DBB DBB IPAddress Address Delegate 1 Delegate 2 Delegate 3 Delegate 4 . . .10.1.0.1 MAC-11 MAC-12 MAC-13 MAC-14 (VLAN#1) 10.1.0.1 MAC-21 MAC-22 . .. (VLAN#2) 10.1.0.1 MAC-31 . . . (VLAN#3)

Table 7 illustrates an example of mapping host IP addresses to aplurality of delegate MAC addresses, e.g., for multiple subnets. Themapping may be sent in a gratuitous group ARP by a DBB to announce itshost IP addresses for each VLAN associated with the DBB. Each delegateMAC address (DBB-MAC1, DBB-MAC2, . . . ) may be mapped to a plurality ofcorresponding host IP addresses in a subnet. Each delegate DBB MACaddress may be associated with a customer or virtual subnet ID for thehost IP addresses. The host IP addresses for each delegate DBB MACaddress may also correspond to a plurality of VLANs (VID-1, VID-2,VID-n, . . . ). The host IP addresses in each subnet may be different.Duplicated host IP addresses, which may be associated with the sameVLANs but with different customer IDs, may be mapped to differentdelegate DBB MAC addresses.

TABLE 7 Information carried by Gratuitous Group ARP DBB VLAN HostDBB-MAC1 VID-1 IP addresses of all hosts in this VLAN (IP Prefix) VID-2IP addresses of all hosts in this VLAN (IP Prefix) VID-n IP addresses ofall hosts in this VLAN (IP Prefix) DBB-MAC2 VID-1 IP addresses of allhosts in this VLAN (IP Prefix) VID-2 IP addresses of all hosts in thisVLAN (IP Prefix) VID-n IP addresses of all hosts in this VLAN (IPPrefix)

FIG. 22 illustrates an embodiment of a fail-over scheme 2200 that may beused for interconnected network districts in a Layer 2 bridged network.The fail-over scheme 2100 may be used in the case any of the DBBs (e.g.,a ToR switch) in the interconnected districts fails. The Layer 2 bridgednetwork may comprise a plurality of core bridges 2212 and a plurality ofDBBs 2222 or district boundary switches in a core district 1810, and aplurality of districts 2220. The districts 2220 may comprise the DBBs2222, a plurality of intermediate switches 2224, and a plurality ofend-stations 2226, e.g., servers/VMs. The Layer 2 bridged network mayalso comprise a DS (not shown) that may be connected to the DBBs 2222,e.g., via the core district 2210. Some of the DBBs 2222, intermediateswitches 2224, and end-stations 2226 may belong to a VLAN established inthe Layer 2 bridged network. The components of the Layer 2 bridgednetwork may be arranged as shown in FIG. 22.

When an active DBB 2222 fails in a VLAN, the VLAN may be establishedusing one or more standby DBBs 2222. The standby DBBs 2222 may establishactive connections with at least some of the intermediate switches 2224that belong to the VLAN and possibly with a new core bridge 2212. Thisis indicated by the dashed lines in FIG. 22. As such, the paths to theend-stations 2226 of the VLAN may not be lost which allows theend-stations 2226 to communicate over the VLAN. When the DBB 2222 in theVLAN fails, the DS may be notified of the failure, for instance bysending an explicit message to the DS or using a keep-alive method.Thus, a DBB may replace the address information of the failed DBB andpossibly other original DBBs 2222 in the VLAN in the entries of the DSaddress table with information of the new DBBs 2222 that were on standbyand then used to replace the failed and other original DBBs 2222. Areplaced failed and original DBB are indicated by circles in FIG. 22.Upon detecting the failed DBB 2222, a replacement DBB may send a LSA tothe DS or the core district 2010 to indicate that the failed DBB'saddresses, including all delegate addresses, are reachable by thereplacement DBB 2222.

With server virtualization, a physical server may host more VMs, e.g.,tens to hundreds of virtual end-stations or VMs. This may result in asubstantial increase in the number of virtual hosts in a DC. Forexample, for a relatively large DC with about 50,000 servers, which mayeach support up to about 128 VMs, the total number of VMs in the DC maybe equal to about 50,000×128 or about 6,400,000 VMs. To achieve dynamicallocation of resources across such large server pool, Ethernet-basedLayer 2 networks may be used in DCs. Such a large Layer 2 network withpotentially a substantial number of virtual hosts may pose newchallenges to the underlying Ethernet technology. For instance, oneissue may be MAC forwarding table scalability due to the flat MACaddress space. Another issue may be handling a broadcast storm caused byARP and other broadcast traffic.

One approach to reduce the size of the MAC forwarding table, alsoreferred to herein as a FDB, in the core of the network may be usingnetwork address encapsulation, e.g., according to IEEE 802.1ah andTRILL. The network address encapsulations of 802.1ah and TRILL aredescribed in IEEE P802.1ah/D4.2 standard and IETF draftdraft-ietf-trill-rbridge-proto1-12-txt, respectively, both of which areincorporated herein by reference. With network address encapsulation,the number of FDB entries in core switches may be reduced to the totalnumber of switches (including edge and core) in the network, independentof the number of VMs. For example, with about 20 servers per edgeswitch, the number of edge switches in a network of about 50,000 serversmay be equal to about 50,000/20 or about 2,500. However, with data pathMAC address learning, the FDB size of edge switches (e.g., ToR switchesin DCs) may be about the same as when network address encapsulation isnot used, which may be substantially large.

Even with selective MAC learning at ToR switches, the FDB size may stillbe substantially large. For example, if a ToR switch has about 40downstream ports, a pair of ToR switches may have up to about 40dual-homed servers connected to the ToR switches. If a server supportsup to about 128 VMs, a ToR switch may have about 128×40/2 or about 2,560VMs connected to the ToR switch in normal operation, e.g., when the TORswitches handle about the same number of VMs. The number of VMs mayincrease to about 5,120 if one ToR switch fails. If each VM communicateson average with about 10 remote VMs simultaneously, the ToR switch FDBsize (e.g., number of entries) may be at least proportional to about2,560 (local VMs)+2,560×10 (remote VMs)+2,500 (ToR switches) or about30,660 entries, which may be further doubled in the failure scenario.

The network address encapsulations in 802.1ah and TRILL may besymmetric. Specifically, the same switches, such as edge switches, mayperform the address encapsulation. The problem with the symmetricnetwork address encapsulations in 802.1ah and TRIL is that an edgeswitch needs to keep track of the remote VMs that communicate with localVMs. The number of the remote VMs may vary substantially. One solutionproposed by A. Greenberg et al. in a paper entitled “Towards a NextGeneration Data Center Architecture: Scalability and Commoditization”,published in PRESTO 08, which is incorporated herein by reference, is tomove the network address encapsulation procedure inside the VMs, thusreducing the switch FDB size to its minimum, which may be equal to thesum of the number of local VMs and the number of edge switches in thenetwork (e.g., equal to about 2,560+2,500 or about 5,060 entries in theabove example). A drawback of this approach is the change of guestoperation system (OS) protocol stack.

Instead, moving the network address encapsulation to a virtual switch ofa physical server (e.g., inside a hypervisor) may reduce the edge switchFDB size and avoid changing the guest OS protocol stack, as describedfurther below. Such a network address encapsulation is referred toherein as asymmetric network address encapsulation since addressdecapsulation is still done elsewhere in edge switches. This mechanismof asymmetric network address encapsulation may reduce the amount ofaddresses maintained in the FDBs of intermediate/edge switches orrouters.

The asymmetric network address encapsulation scheme may be implementedin a Layer 2 network that comprises edge and core switches, such as inthe different network embodiments described above. For instance, theedge switches may correspond to ToR switches in DCs. Each edge switchmay be assigned a unique ID, which may be a MAC address (as in 802.1ah),an about 16 bit nickname (as in TRILL), or an IP address. The networkmay be configured to forward a frame based on the destination edgeswitch ID carried in the header of the frame from an ingress edge switchto the egress edge switch. The frame may be forwarded inside the networkusing any transport technology. The asymmetric network addressencapsulation scheme may be similar to the address encapsulation schemein 802.1ah, also referred as MAC-in-MAC. MAC learning may be disabled inthe network but enabled on the edge switch server facing ports. Theterms server, end-station, and host may be used interchangeably herein.The terms virtual server, VM, virtual end-station, and virtual host mayalso be used interchangeably herein.

In MAC-in-MAC, there are two types of MAC addresses: the MAC addressesassigned to edge switches, also referred to as network addresses orbackbone MAC (B-MAC) addresses, and the MAC addresses used by VMs, alsoreferred to as customer MAC (C-MAC) addresses. FIG. 23 illustrates anembodiment of a typical physical server 2300, which may be a dual-homedserver in a DC. The physical server 2300 may comprise a virtual switch2310, a plurality of VMs 2340, and a plurality of physical NetworkInterface Cards (pNICs) 2350. The virtual switch 2310 may comprise anARP proxy 2330 and a FDB 2320, which may comprise a local FDB 2322 and aremote FDB 2324. The virtual switch 2310 may be located inside ahypervisor of the physical server 2300. The virtual switch 2310 may beconnected to the VMs via a plurality of corresponding virtual NetworkInterface Cards (NICs) 2342 of the VMs 2340 and a plurality ofcorresponding virtual switch ports 2312 of the virtual switch 2310. Thevirtual switch 2310 may also be connected to the pNICs 2350 via aplurality of corresponding virtual switch trunk ports 2314 of thevirtual switch 2310. The pNICs 2350 may serve as uplinks or trunks forthe virtual switch 2310. The physical server 2300 may be connected to aplurality of edge switches 2360 via corresponding pNICs 2350 of thephysical server 2300. Thus, the edge switches 2360 may be connected viathe components of the physical server 2300 (the pNICs 2350 and thevirtual switch 2310) to the VMs 2340. The components of the physicalserver 2300 may be arranged as shown in FIG. 23.

For load balancing, traffic may be distributed to the trunks (pNICs2350) based on the virtual port IDs or VM source C-MAC addresses of thetraffic. Each VM 2340 may have a virtual NIC 2342 with a uniquelyassigned C-MAC address. A VM 2340 may send traffic to an edge switch2360 during normal operation. For example, a first VM 2340 (VM1) maysend a plurality of frames intended to external VMs in other physicalservers in the network (not shown) via a corresponding first edge switch2350 (edge switch X). A second edge switch 2360 (edge switch R) may be abackup for edge switch X. When edge switch X becomes unreachable due toa failure (e.g., the corresponding pNIC 2350 fails, the link between thepNIC 2350 and edge switch X fails, or edge switch X fails), the virtualswitch 2310 may then send the frames to edge switch R.

In the FDB 2320, the local FDB 2322 may correspond to the local VMs (VMs2340) and may comprise a plurality of C-MAC destination addresses (C-MACDAs), a plurality of VLAN IDs, and a plurality of associated virtualswitch port IDs. The C-MAC DAs and VLAN IDs may be used to look up thelocal FDB 2322 to obtain the corresponding virtual switch port IDs. Theremote FDB 2324 may correspond to external VMs (in other physicalservers) and may comprise a plurality of B-MAC destination addresses(B-MAC DAs) and a plurality of C-MAC DAs associated with the B-MAC DAs.The C-MAC DAs may be used to look up the remote FDB 2324 by the localVMs to obtain the corresponding B-MAC DAs. The remote FDB 2324 may bepopulated by the ARP proxy 2330, as described below.

Based on the symmetric address encapsulation, an Ethernet frame from aVM 2340 may be untagged or tagged. If the frame is untagged, the VLAN IDassigned to the corresponding virtual switch port 2312 may be used. Inthe upstream direction from the VM 2340 to an edge switch 2360, thevirtual switch 2310 may perform the following steps after receiving anEthernet frame from the VM 2340:

Step 1: Use C-MAC DA and VLAN ID in the table lookup of the local FDB2322. If a match is found, forward the frame to the virtual switch port2312 that is specified in the matched FDB entry (by the virtual switchport ID). Else, go to step 2.

Step 2: Use C-MAC DA in the table lookup of the remote FDB 2324. If amatch is found, perform a MAC-in-MAC encapsulation based asymmetricnetwork address encapsulation (described below) and forward the frame tothe virtual switch trunk port 2314 that is associated with the C-MAC SAin the frame. Else, go to step 3.

Step 3: Discard the frame and send an enhanced ARP request to an ARPserver in the network (not shown).

FIG. 24 illustrates an embodiment of an asymmetric network addressencapsulation scheme 2400 that may be used in the physical server. Basedon the asymmetric network address encapsulation scheme 2400, a VM 2402may send, in the upstream direction, a frame intended to anotherexternal or remote VM in another physical server in the network (notshown). The frame may comprise a C-MAC DA (B) 2410 of the remote VM, aC-MAC SA (A) 2412 of the VM 2402, a C-VLAN ID 2414 for the VLAN of theVM 2402, data or payload 2416, and a Frame Check Sequence (FCS) 2418.The VM 2402 may send the frame to a virtual switch 2404.

The virtual switch 2404 (in the same physical server) may receive theframe from the VM 2402. The virtual switch 2404 may process the frameand add a header to the frame to obtain a MAC-in-MAC frame. The headermay comprise a B-MAC DA (Y) 2420, a B-MAC SA (0) 2422, a B-VLAN ID 2424,and an Instance Service ID (I-SID) 2426. The B-MAC address (Y) may beassociated with the C-MAC DA (B) 2410 in an edge switch 2406. The B-MACaddress (Y) may indicate the location of the remote VM that has theC-MAC address (B). The B-MAC SA 2422 may be set to zero by the virtualswitch 2404. The B-VLAN ID 2424 may be set to the C-VLAN ID 2414. TheI-SID 2426 may be optional and may not be used in the header if theEthernet frame is only sent to the C-MAC DA (B). The virtual switch 2404may then send the MAC-in-MAC frame to the edge switch 2406.

The edge switch 2406 (connected to the physical server) may receive theMAC-in-MAC frame from the virtual switch 2404. The edge switch 2406 mayprocess the header of the MAC-in-MAC frame to obtain a new header in theMAC-in-MAC frame. The new header may comprise a B-MAC DA (Y) 2440, aB-MAC SA (X) 2442, a B-VLAN ID 2444, and an I-SID 2446. The B-MAC SA (X)2442 may be set to the B-MAC address (X) of the edge switch 2406. TheB-VLAN ID 2444 may be changed if necessary to match a VLAN in thenetwork. The remaining fields of the header may not be changed. The edgeswitch 2406 may then forward the new MAC-in-MAC frame based on the B-MACDA (Y) 2440 and possibly the B-VAN ID 2444 via the network core 2408,e.g., a core network or a network core district.

In the downstream direction, the edge switch 2406 may receive aMAC-in-MAC frame from the network core 2408 and perform a framedecapsulation. The MAC-in-MAC frame may comprise a header and anoriginal frame sent from the remote VM to the VM 2402. The header maycomprise a B-MAC DA (X) 2460 for the edge switch 2406, a B-MAC SA (Y)2462 that corresponds to remote VM and the edge switch 2406, a B-VLAN ID2464 of the VLAN of the remote VM, and an I-SID 2466. The original framefor the remote VM may comprise a C-MAC DA (A) 2470 for the VM 2402, aC-MAC SA (B) 2472 of the remote VM, a C-VLAN ID 2474 associated with theVM 2402, data or payload 2476, and a FCS 2478. The edge switch 2406 mayremove the header from the MAC-in-MAC frame and forward the remainingoriginal frame to the virtual switch 2404. The edge switch 2406 may lookup its forwarding table using C-MAC DA (A) 2470 and C-VLAN ID 2474 toget an outgoing switch port ID and forward the original frame out on thephysical server facing or connected to the corresponding switch port. Inturn, the virtual switch 2404 may forward the original frame to the VM2402. The virtual switch 2404 may forward the original frame to the VM2402 based on the C-MAC DA (A) 2470 and the C-VLAN ID 2474.

The forwarding tables in the edge switch 2406 may include a local FDBand a remote FDB. The local FDB may be used for forwarding frames forlocal VMs and may be populated via MAC learning and indexed by the C-MACDA and C-VLAN ID in the received frame. The remote FDB may be used forforwarding frames to remote VMs and may be populated by a routingprotocol or a centralized control/management plane and indexed by theB-MAC DA and possibly the B-VLAN ID in the received frame.

In the asymmetric address encapsulation scheme 2400, the MAC-in-MACencapsulation may be performed at the virtual switch 2404, while theMAC-in-MAC de-capsulation may be performed at the edge switch 2406. Assuch, the FDB size in the edge switches may be substantially reduced andbecome more manageable even for a substantially large Layer 2 network,e.g., in a mega DC. The remote FDB size in the virtual switch 2404 maydepend on the number of remote VMs in communication with the local VMs,e.g., the VM 2402. For example, if a virtual switch supports about 128local VMs and each local VM on average communicates with about 10 remoteVMs concurrently, the remote FDB may comprise about 128×10 or about1,289 entries.

FIG. 25 illustrates an embodiment of an ARP processing scheme 2500 thatmay be used in the physical server 2300. Based on the ARP processingscheme 2500, a VM 2502 may broadcast an ARP request for a remote VM. TheARP request may comprise a C-MAC DA (BC) 2510 that indicates a broadcastmessage, a C-MAC SA (A) 2512 of the VM 2502, a C-VLAN ID 2514 for theVLAN of the VM 2502, ARP payload 2516, and a FCS 2518.

A virtual switch 2504 (in the same physical server), which may beconfigured to intercept all ARP messages from local VMs, may interceptthe ARP request for a remote VM. An ARP proxy in the virtual switch 2504may process the ARP request and add a header to the frame to obtain aunicast extended ARP (ERAP) message. The frame may be encapsulated usingMAC-in-MAC, e.g., similar to the asymmetric network addressencapsulation scheme 2400. The header may comprise a B-MAC DA 2520, aB-MAC SA (0) 2522, a B-VLAN ID 2524, and an I-SID 2526. The B-MAC DA2520 may be associated with an ARP server 2508 in the network. TheB-VLAN ID 2524 may be set to the C-VLAN ID 2514. The I-SID 2526 may beoptional and may not be used. The EARP message may also comprise a C-MACDA (Z) 2528, a C-MAC SA (A) 2530, a C-VLAN ID 2532, an EARP payload2534, and a FCS 2536. The ARP proxy may replace the C-MAC DA (BC) 2510and the ARP payload 2516 in the received frame with the C-MAC DA (Z)2528 for the remote VM and the EARP payload 2534, respectively, in theEARP message. The virtual switch 2504 may then send the EARP message tothe edge switch 2506.

The edge switch 2506 may process the header in the EARP message toobtain a new header. The new header may comprise a B-MAC DA (Y) 2540, aB-MAC SA (X) 2542, a B-VLAN ID 2544, and an I-SID 2546. The B-MAC SA (X)2542 may be set to the B-MAC address (X) of the edge switch 2506. TheB-VLAN ID 2544 may be changed if necessary to match a VLAN in thenetwork. The remaining fields of the header may not be changed. The edgeswitch 2506 may then forward the new EARP message to the ARP server 2508in the network.

The ARP server 2508 may process the received EARP message and return anEARP reply to the edge switch 2506. The EARP reply may comprise a headerand an ARP frame. The header may comprise a B-MAC DA (X) 2560 for theedge switch 2506, a B-MAC SA 2562 of the ARP server 2508, a B-VLAN ID2564, and an I-SID 2566. The ARP frame may comprise a C-MAC DA (A) 2568for the VM 2502, a C-MAC SA (Z) 2570 for the requested remote VM, aC-VLAN ID 2572, an EARP payload 2574, and a FCS 2576. The edge switch2506 may decapsulate the EARP message by removing the header and thenforward the ARP frame to the virtual switch 2504. The virtual switch2504 may process the ARP frame and send an ARP reply accordingly to theVM 2502. The ARP reply may comprise a C-MAC DA (A) 2590 for the VM 2502,a C-MAC SA (B) 2592 associated with remote VM's location, a C-VLAN ID2594, an ARP payload 2596, and a FCS 2598.

The ARP proxy in the virtual switch 2504 may also use the EARP messageto populate the remote FDB in the edge switch 2506. The ARP proxy maypopulate an entry in the FDB table with a remote C-MAC and remote switchB-MAC pair, which may be found in the EARP payload 2574. The C-MAC andremote switch B-MAC may be found in a sender hardware address (SHA)field and a sender location address (SLA) field, respectively, in theEARP payload 2574.

A hypervisor in the physical server that comprises the virtual switch2504 may also register a VM, e.g., the local VM 2502 or a remote VM,with the ARP server 2508 in a similar manner of the ARP processingscheme 2500. In this case, the virtual switch 2504 may send a unicastEARP frame to the ARP server 2508 with all the sender fields equal toall the target fields. Another way to register the VM is described inU.S. Provisional Patent Application No. 61/389,747 by Y. Xiong et al.entitled “A MAC Address Delegation Scheme for Scalable Ethernet Networkswith Duplicated Host IP Addresses,” which is incorporated herein byreference as if reproduced in its entirety. This scheme may handle theduplicated IP address scenario.

FIG. 26 illustrates an embodiment of an EARP payload 2600 that may beused in the ARP processing scheme 2500, such as the EARP payload 2574.The EARP payload 2600 may comprise a hardware type (HTYPE) 2610, aprotocol type (PTYPE) 2612, a hardware address length (HLEN) 2614, aprotocol address length (PLEN) 2616, an operation field (OPER) 2618, aSHA 2620, a sender protocol address (SPA) 2622, a target hardwareaddress (THA) 2624, and a target protocol address (TPA) 2626, which maybe elements of a typical ARP message. Additionally, the EARP payload2600 may comprise a SLA 2628 and a target location address (TLA) 2630.FIG. 26 also shows the bit offset for each field in the EARP payload2600, which also indicates the size of each field in bits.

One issue with using the ARP server (e.g., the ARP server 2508) anddisabling MAC learning in the network is the case where a VM becomesunreachable due to a failure of its edge switch or the link connectingthe ARP server to the edge switch. In this case, it may take some timefor the virtual switch to know the new location of a new or replacementedge switch for the VM. For example, if the edge switch X in thephysical server 2300 becomes unreachable, the virtual switch 2310 mayforward frames from VM1 to the edge switch R, which may become the newlocation for VM1.

To reduce the time for updating the remote FDB in a virtual switch 2310about the new location of a VM, a gratuitous EARP message may be used.The virtual switch 2310 may first send a gratuitous EARP message to theedge switch R in a MAC-in-MAC encapsulation frame, including a B-MAC DAset to broadcast address (BC). In the gratuitous EARP message, the SHA(e.g., SHA 2620) may be set equal to the THA (e.g., THA 2624), the SPA(e.g., SPA 2622) may be set equal to the TPA (e.g., TPA 2626), and theSLA (e.g., SLA 2628) may be set equal to TLA (e.g., TLA 2630). The edgeswitch R may then send the gratuitous EARP message to a plurality of orto all other edge switches in the network, e.g., via a distributiontree. When an edge switch receives the gratuitous EARP message, the edgeswitch may decapsulate the message and send the message out on the edgeswitch's server facing ports. When a virtual switch then receives thegratuitous EARP message, the virtual switch may update its remote FDB ifthe SHA already exists in the remote FDB. The ARP server in the networkmay update the new location of the affected VM in the same way.

The asymmetric network address encapsulation scheme described above mayuse the MAC-in-MAC encapsulation in one embodiment. Alternatively, thisscheme may be extended to other encapsulation methods. If TRILL issupported and used in a network, where an edge switch is identified byan about 16 bit nickname, the TRILL encapsulation may be used in theasymmetric network address encapsulation scheme. Alternatively, anIP-in-IP encapsulation may be used if an edge switch is identified by anIP address. Further, network address encapsulation may be performed atthe virtual switch level and the network address de-capsulation may beperformed at the edge switch level. In general, the network addressencapsulation scheme may be applied at any level or any of the networkcomponents as long as the encapsulation and de-capsulation are kept atdifferent levels or components.

In a bridged network that is partitioned into districts, such as in theinterconnected network districts 1800, a DBB may be a bridgeparticipating in multiple districts. The DBB's address may be referredto herein as a network address to differentiate the DBB's address fromthe C-MAC addresses of the VMs in each district. Using the asymmetricaddress encapsulation scheme above, the encapsulation of the networkaddress may be performed at the switch closer to hosts or the virtualswitch closer to virtual hosts. For example, the intermediate switches1824, e.g., ToR switches, may perform the network address encapsulation.The intermediate switches 1824 may encapsulate the data frames comingfrom the subsets of hosts and that comprise a target DBB address.However, the intermediate switches 1824 may not alter data framesincoming from the network side, e.g., the DBBs 1822 in the core district1810. The target DBB 1822, which is one level above the intermediateswitch 1824, may decapsulate the data frames from network side (coredistrict 1810) and forward the decapsulated data frame towards hostswithin its district.

In an embodiment, a virtual switch inside a physical server (e.g., anend-station 1826) may perform the network address encapsulation, whilethe target DBB 1822 may perform the network address decapsulation. Inthis case, the DBB 1822 that performs the decapsulation may be twolevels above the virtual switch (in the end-station 1826) that performsthe encapsulation.

The bridged network connected to the DBB 1822 (e.g., the core district1810) may be IP based. The core network (or district) that interconnectsthe DBBs may be a L3 Virtual Private Network (VPN), a L2 VPN, orstandard IP networks. In such scenarios, the DBB may encapsulate the MACdata frames from its local district with a proper target DBB address,which may be an IP or MPLS header.

FIG. 27 illustrates an embodiment of a data frame forwarding scheme 2700that may be used in a Layer 2 bridged network, such as for theinterconnected network districts 1800. The data frame forwarding scheme2700 may also implement the asymmetric network address encapsulationscheme above. The Layer 2 bridged network may comprise a core district2710, a plurality of DBBs 2722 or district boundary switches in aplurality of districts 2720 connected to the core district 2710, and aplurality of intermediate or edge switches 2724 and physical servers2726 connected to corresponding DBBs 2722 in their districts 2720. Thephysical servers 2726 may comprise a plurality of VMs and virtualswitches (not shown). Some of the DBBs 2722, intermediate/edge switches2724, and physical servers 2726 across the districts 2720 may belong toa VLAN established in the Layer 2 bridged network and associated with aVLAN ID. The components of the Layer 2 bridged network may be arrangedas shown in FIG. 27.

According to the asymmetric network address encapsulation scheme, anintermediate/edge switch 2724 may receive a frame 2740, e.g., anEthernet frame, from a first VM (host A) in a physical server 2726 in afirst district (district 1). The frame 2740 may be intended for a secondVM (host B) in a second physical server 2726 in a second district(district 2). The frame 2740 may comprise a B-MAC DA 2742 for a secondDBB (DBB2) in district 2, a B-MAC SA 2744 for host A (ToR A), a C-MAC DA2746 for host B (B), a C-MAC SA 2748 for host A (A), an IP-SA 2750 forhost A (A), an IP-DA 2752 for host B (B), and payload. Theintermediate/edge switch 2724 may forward the frame 2740 to a first DBB2722 (DBB1) in district 1. DBB1 may receive and process the frame 2740to obtain an inner frame 2760. The inner frame 2760 may comprise a B-MACDA 2762 for DBB2, a B-MAC SA 2764 for DBB1, a C-MAC DA 2766 for host B(B), a C-MAC SA 2768 for host A (A), an IP-SA 2770 for host A (A), anIP-DA 2772 for host B (B), and payload. DBB1 may then forward the innerframe 2760 to district 2 via the core district 2710.

DBB2 in district 2 may receive and decapsulate the inner frame 2760 toobtain a second frame 2780. DBB2 may remove B-MAC DA 2762 for DBB2 and aB-MAC SA 2764 from the inner frame 2760 to obtain the second frame 2780.Thus, the second frame 2780 may comprise a C-MAC DA 2782 for host B (B),a C-MAC SA 2784 for host A (A), an IP-SA 2786 for host A (A), an IP-DA2788 for host B (B), and payload. DBB2 may send the second frame 2780 tohost B in district 2.

In the data frame forwarding scheme 2700, the intermediate/edge switch2724 may not perform the MAC-in-MAC function for frames received fromlocal physical servers 2726 connected to the intermediate/edge switch2724. In another embodiment, the encapsulation procedure of the firstframe 2740 may be performed by a virtual switch in the physical server2726 instead of the intermediate/edge switch 2724, which may forward thefirst frame 2740 without processing from the physical server 2726 to thecorresponding DBB 2722.

FIG. 28 illustrates an embodiment of an enhanced ARP processing method2800 that may be used in a Layer 2 bridged network, such as for theinterconnected network districts 1800. The enhanced ARP processingmethod 2800 may begin at step 2801, where a local host 2810 may send anARP request to a local location 2830 via a first bridge 2820, e.g., alocal DBB. The local location 2830 may correspond to the same locationor district as the local host 2810. The ARP request may be sent toobtain a MAC address associated with a remote host 2860. The local host2810 may be assigned an IP address IPA and a MAC address A. The remotehost 2860 may be assigned an IP address IPB and a MAC address B. The ARPrequest may comprise a SA MAC address A and a SA IP address IPA for thelocal host 2810. The ARP request may also comprise a DA MAC address setto zero and a DA IP address IPB for the remote host 2860. The locallocation 2830 may forward the ARP request to an ARP server 2840 in thenetwork.

At step 2802, the ARP server 2840 may send an EARP response to the firstbridge 2820. The EARP response may comprise a SA MAC address A and a SAIP address IPA for the local host 2810, a DA MAC address B and a DA IPaddress IPB for the remote host 2860, and a MAC address for a secondbridge in a remote location 2850 of the remote host 2860. At step 2803,the first bridge 2820 may process/decapsulate the EARP response and sendan ARP response to the local host 2810. The ARP response may comprisethe MAC address A and IP address IPA for the local host 2810, and theMAC address B and the IP address IPB for the remote host 2860. Thus, thelocal host 2810 may become aware of the MAC address B of the remote host2860. The first bridge 2820 may also associate (in a local table) theMAC address Y of the remote bridge in the remote location 2850 with theIP address IPB of the remote host 2860. The first bridge 2820 may notneed to store the MAC address B of the remote host 2860.

At step 2804, the local host 2810 may send a data frame intended for theremote host 2860 to the first bridge 2820. The data frame may comprise aSA MAC address and SA IP address of the local host 2810, and the DA MACaddress and DA IP address of the remote host 2860. At step 2805, thefirst bridge 2820 may receive and process/encapsulate the data frame toobtain an inner frame. The inner frame may comprise a SA MAC address Xof the first bridge 2820, a DA MAC address Y of the remote bridge, a DAMAC address B and a DA IP address IPB of the remote host 2860, and a SAMAC address A and a SA IP address IPA of the local host 2810. At step2806, the remote bridge in the remote location 2850 may receive theinner frame and process/decapsulate the inner frame to obtain a secondframe by removing the SA MAC address X of the first bridge 2820 and theDA MAC address Y of the remote bridge. Thus, the second frame may besimilar to the initial frame sent from the local host 2810. The remotebridge may then send the second frame to the remote host 2860. Themethod 2800 may then end.

In the enhanced ARP processing method 2800, the core network may use802.1aq or TRILL for topology discovery. If the core network uses802.1aq for topology discovery, then the first bridge 2820 may notencapsulate the frame sent form the local host 2810 and may forward theframe to the remote location 2850 without processing. Further, the frameforwarded through the core network may be flooded only in the secondlocation 2850 and only when the outbound port indicated in the frame hasnot been learned.

In an embodiment, an extended address resolution scheme may beimplemented by district gateways or gateway nodes that may be TRILL edgenodes, MAC-in-MAC edge nodes, or any other type of overlay network edgenodes. The extended address resolution scheme may be based on the ARPproxy scheme implemented by a DBB in a plurality of districts in a Layer2 bridged network, such as the ARP proxy scheme 1900. For example, theintermediate/edge nodes 2724 that may be connected to a plurality ofphysical servers and/or VMs may implement an extended address resolutionscheme similar to the ARP proxy scheme described above. The gateway nodemay use the DS server in the ARP proxy scheme to resolve mapping betweena target destination (e.g., host) and an egress edge node. The egressedge node may be a target district gateway, a TRILL egress node, aMAC-in-MAC edge node, or any other type of overlay network edge node.The reply from the DS may also be an EARP reply as described above.

The extended address resolution scheme may be used to scale DC networkswith a substantial number of hosts. The overlay network (e.g., bridgednetwork) may be a MAC-in-MAC, TRILL, or other types of Layer 3 or Layer2 over Ethernet networks. The overlay network edge may be a networkswitch, such as an access switch (or ToR switch) or an aggregationswitch (or EoR switch). The overlay network edge may also correspond toa virtual switch in a server. There may be two scenarios for overlaynetworks for using the extended address resolution scheme. The firstscenario corresponds to a symmetric scheme, such as for TRILL orMAC-in-MAC networks. In this scenario, the overlay edge node may performboth the encapsulation and decapsulation parts. The second scenariocorresponds to an asymmetric scheme, where the overlay network mayimplement the asymmetric network address encapsulation scheme above.

FIG. 29 illustrates an embodiment of an extended address resolutionmethod 2900 that may be implemented in an overlay network. The extendedaddress resolution method 2900 may begin at step 2901, where a first VM2910 (VM A) may send a frame or packet addressed for a second VM 2980(VM B) to a first hypervisor (HV) 2920 (HV A). VM A and VM B may be endhosts in different districts. VM A may be connected to HV A in a firstdistrict and VM B may be connected to a second HV 2970 (HV B) in asecond district. The HV may be an overlay network node configured toencapsulate or add the overlay network address header on a data frame orpacket. In the symmetric scheme scenario, the HV may be a DBB, a TRILLedge node, or a MAC-in-MAC edge node. In the asymmetric scheme scenario,the HV may be a virtual switch within a hypervisor or an access switch.

At step 2902, HV A may send an address resolution (AR) request to an ARPserver 2930 to retrieve mapping from VM B IP address to a VM B MACaddress and HV B MAC address pair, in the case of the symmetric scheme.The ARP server may comprise or correspond to a DS server, such as the DS1940. In the asymmetric scheme, the mapping may be from VM B IP addressto a VM B MAC address and second DBB 2960 (DBB B) MAC address pair. DBBB may be a remote DBB in the same district of VM B.

HV A may also be configured to intercept (broadcasted) ARP requests fromlocal VMs and forward the ARP requests to the DS server. HV A may thenretrieve EARP replies from the DS server and cache the mappings betweentarget addresses and target gateway addresses (as indicated by the EARPreplies). The target gateway address may also be referred to herein as atarget location address. In another embodiment, instead of interceptingARP requests by HV A, the DS server may send consolidated mappinginformation to HV A on regular or periodic basis or when VMs move ormigrate between districts. The consolidated mapping information maycomprise the same information exchanged with L2GWs in the virtual Layer2 networks described above. For instance, the consolidated mappinginformation may be formatted as gratuitous group announcements, asdescribed above.

At step 2903, HV A may create an inner address header that comprise (SA:VM A MAC, DA: VM B MAC) and an outer header that comprises (SA: HV AMAC, DA: HV B MAC), in the case of the symmetric scheme. In theasymmetric scheme, the outer header may comprise (SA: HV A MAC, DA: DBBB MAC). HV A may add the inner header and outer header to the framereceived from VM A and send the resulting frame to a bridge 2940connected to HV A in the same district. Within the district, the DA ofthe outer header, which may be HV B MAC or DBB B MAC, may not be known.

At step 2904, the frame may be forwarded from the bridge 2940 to a firstDBB 2950 (DBB A) in the district. At DBB A, the DA HV B MAC or DBB B MACmay be known since the core may be operating on routed forwarding (e.g.,802.1aq SPBM or TRILL) and learning may be disabled in the core. At step2905, DBB A may forward the frame to DBB B.

At step 2906, DBB B may forward the frame to HV B since DBB may know allHV addresses from the routing subsystem, in the case of the symmetricscheme. In the asymmetric scheme, DBB may remove the outer headercomprising (DA: DBB MAC) and forward the frame to VM B MAC in theremaining header, since addresses local to the district may beregistered and known within the district.

At step 2907, HV B may receive the frame, remove the outer headercomprising (DA: HV B MAC), and forward the resulting frame to VM B MACin the remaining header, since addresses local to the server are knownto HV B, in the case of the symmetric scheme. Additionally, HV B maylearn the mapping from VM A MAC (SA in the remaining header) to HV A MAC(SA in the removed header), which may be subsequently used in replyframes from VM B to VM A. In the asymmetric scheme, in addition toforwarding the frame to VM B, HV B may send an ARP message to the ARP(or DS) server 2930 to retrieve the mapping from VM A MAC (SA in theremaining header) to DBB A MAC, which may be subsequently used in replyframes from VM B to VM A.

VM B may then send frames addressed to VM A (IP destination address). Atstep 2908, HV B may create an inner address header that comprises (SA:VM B MAC, DA: VM A MAC) and an outer header that comprises (SA: HV BMAC, DA: HV A MAC) to a frame, in the case of the symmetric scheme. HV Bmay maintain VM A IP to VM A MAC mapping and VM A MAC to HV A MACmapping from a previously received message or AR response. In theasymmetric scheme, the outer header may comprise (SA: HV B MAC, DA: DBBA MAC). HV B may maintain VM A MAC to DBB A MAC mapping from apreviously received AR response. Alternatively, HV B may send an ARPmessage to the ARP (or DS) server to retrieve the mapping when needed.The frame may then be forwarded from VM B to VM A in the same mannerdescribed in the steps above (e.g., in the reverse direction). Themethod 2900 may then end.

FIG. 30 illustrates an embodiment of a network component unit 3000,which may be any device that sends/receives packets through a network.For instance, the network component unit 3000 may be located at theL2GWs across the different locations/domains in the virtual/pseudo Layer2 networks. The network component unit 3000 may comprise one or moreingress ports or units 3010 for receiving packets, objects, or TLVs fromother network components, logic circuitry 3020 to determine whichnetwork components to send the packets to, and one or more egress portsor units 3030 for transmitting frames to the other network components.

The network components described above may be implemented on anygeneral-purpose network component, such as a computer system or networkcomponent with sufficient processing power, memory resources, andnetwork throughput capability to handle the necessary workload placedupon it. FIG. 31 illustrates a typical, general-purpose computer system3100 suitable for implementing one or more embodiments of the componentsdisclosed herein. The general-purpose computer system 3100 includes aprocessor 3102 (which may be referred to as a CPU) that is incommunication with memory devices including second storage 3104, readonly memory (ROM) 3106, random access memory (RAM) 3108, input/output(I/O) devices 3110, and network connectivity devices 3112. The processor3102 may be implemented as one or more CPU chips, or may be part of oneor more application specific integrated circuits (ASICs).

The second storage 3104 is typically comprised of one or more diskdrives or tape drives and is used for non-volatile storage of data andas an over-flow data storage device if RAM 3108 is not large enough tohold all working data. Second storage 3104 may be used to store programsthat are loaded into RAM 3108 when such programs are selected forexecution. The ROM 3106 is used to store instructions and perhaps datathat are read during program execution. ROM 3106 is a non-volatilememory device that typically has a small memory capacity relative to thelarger memory capacity of second storage 3104. The RAM 3108 is used tostore volatile data and perhaps to store instructions. Access to bothROM 3106 and RAM 3108 is typically faster than to second storage 3104.

While several embodiments have been provided in the present disclosure,it should be understood that the disclosed systems and methods might beembodied in many other specific forms without departing from the spiritor scope of the present disclosure. The present examples are to beconsidered as illustrative and not restrictive, and the intention is notto be limited to the details given herein. For example, the variouselements or components may be combined or integrated in another systemor certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described andillustrated in the various embodiments as discrete or separate may becombined or integrated with other systems, modules, techniques, ormethods without departing from the scope of the present disclosure.Other items shown or discussed as coupled or directly coupled orcommunicating with each other may be indirectly coupled or communicatingthrough some interface, device, or intermediate component whetherelectrically, mechanically, or otherwise. Other examples of changes,substitutions, and alterations are ascertainable by one skilled in theart and could be made without departing from the spirit and scopedisclosed herein.

What is claimed is:
 1. An apparatus comprising: a service network; and aplurality of Layer 2 sites connected by the service network via aplurality of gateways, wherein the gateways are configured to map aplurality of Internet Protocol (IP) addresses of a plurality of hostsunder a plurality of virtual local area networks (VLANs) in theplurality of Layer 2 sites to a plurality of addresses of correspondingother gateways, inform the other gateways in other Layer 2 sites of theIP addresses mapped under each of the VLANs in local Layer 2 sites, andforward data frames originated from the hosts in the local Layer 2 sitesto the other gateways in the other Layer 2 sites when destinations ofthe data frames are residing in the other Layer 2 sites.
 2. Theapparatus of claim 1, wherein the gateways are configured to send theother gateways in the other Layer 2 sites of updates of the mapping oftheir own MAC addresses and the IP addresses of hosts under each of theVLANs in the Local Layer 2 sites of the gateways when changes, includinghosts being added, deleted, and/or moved, occur to update the othergateways in an incremental manner, and wherein the gateways areconfigured to keep only the mapping for the VLAN that are active in thelocal Layer 2 sites when the gateways receive updates of the mapping ofgateway MAC and IP address under each of the VLANs from the othergateways.
 3. The apparatus of claim 1, wherein the hosts and a pluralityof switches in the Layer 2 sites are not aware of the MAC addresses ofthe hosts that run IP based applications in other Layer 2 sites evenwhen all the hosts in all the Layer 2 sites belong to a single Layer 2network.
 4. The apparatus of claim 1, wherein the gateways are not awareof the MAC addresses of the hosts that run IP based applications inother Layer 2 sites even when all the hosts in all the Layer 2 sitesbelong to a single Layer 2 network.
 5. The apparatus of claim 4, whereinwhen a host sends out one or more Address Resolution Protocol(ARP)/Neighbor Discovery (ND) requests to obtain a MAC address ofanother target host in another Layer 2 site, the host receives a MACaddress of a gateway associated with the other Layer 2 site, where thetarget host resides under the same VLAN, as the response to the ARP/NDrequests.
 6. The apparatus of claim 5, wherein a host sends the ARP/NDrequests to obtain the MAC address of another target host in the otherLayer 2 site in the same VLAN as the requesting hosts, and wherein theMAC address of the gateway associated with the other Layer 2 site isreturned to the host when the target hosts in the same VLAN exists inthe other site.
 7. The apparatus of claim 1, wherein the gateways areconfigured to maintain a plurality of MAC addresses for a plurality ofhosts that run non-IP applications under each of the VLANs in otherLayer 2 sites, wherein the hosts and a plurality of local switches inthe Layer 2 sites are not aware of the MAC addresses of the hosts thatrun non-IP applications even when all the hosts in all the Layer 2 sitesbelong to a single Layer 2 network, and wherein the gateways are awareof the MAC addresses of the hosts that run non-IP applications in remotesites.
 8. The apparatus of claim 7, wherein the hosts are configured tosend out a plurality of Address Resolution Protocol (ARP)/NeighborDiscovery (ND) requests to obtain a plurality of MAC addresses of otherhosts that run non-IP applications in other Layer 2 sites and receivethe MAC addresses of the other hosts that run non-IP addresses inresponse to the ARP/ND requests.
 9. The apparatus of claim 1, whereinthe service network is a service provider network, a core network, aLayer 3 network, a Layer 2.5 network, or a Layer 2 network.
 10. Theapparatus of claim 1, wherein the gateways are Layer 2 gateways, whereinthe hosts run a plurality of applications that are instantiated onservers and/or virtual machines, and wherein the Layer 2 sites arelocated within one data center (DC), over multiple DCs, or over multiplebuildings and/or floors.
 11. The apparatus of claim 1, wherein thegateways are Layer 2 gateways, wherein the hosts run a plurality ofapplications, instantiated on servers and/or virtual machines, andbelong to one Layer 2 network that is located in multiple sites.
 12. Theapparatus of claim 1, wherein the gateways maintain local hostsinformation tables for the local hosts in the same Layer 2 sites of thegateways, and wherein the local hosts information tables comprise amapping of IP Address to a MAC address under each VLAN for each of thelocal hosts.
 13. The apparatus of claim 12, wherein the gateways sendout a plurality of aggregated IP addresses of the local hosts under eachVLAN to the other gateways in the other Layer 2 sites, and wherein theaggregated IP addresses' entries are substantially fewer than thecorresponding local hosts information tables' entries.
 14. The apparatusof claim 12, wherein the gateways send out a plurality of requests toother gateways in other Layer 2 sites to solicit a plurality of IPaddresses in aggregated form under a VLAN.
 15. The apparatus of claim 1,wherein the plurality of addresses comprise one of a plurality of MACaddresses and a plurality of IP addresses.
 16. A network componentcomprising: a receiver configured to receive via a service network aframe encapsulated using one or two layers of Ethernet headers thatcomprise: a destination address in an outer Ethernet header destined toa local gateway of a local Layer 2 site from a remote host in a remoteLayer 2 site; and a type field that indicates whether the outer Ethernetheader requires decapsulation or another type field in the Ethernetheader that indicates whether the destination address in the Ethernetheader requires translation; a logic circuit configured to translate thedestination address in the Ethernet header to a Media Access Control(MAC) address of a local host based on an Ethernet payload's InternetProtocol (IP) destination address of the local host in the local Layer 2site when the destination address requires translation; and atransmitter configured to send to another gateway of another Layer 2site a frame originated from the local host in the local Layer 2 siteconnected to the other Layer 2 site via the service network when thedestination host resides in the other Layer 2 site.
 17. The networkcomponent of claim 16, wherein when the local host sends out an AddressResolution Protocol (ARP)/Neighbor Discovery (ND) request to obtain aMAC address for a target host in a remote Layer 2 site, a MAC address ofa corresponding remote gateway in the remote Layer 2 site is returned inan ARP/ND response.
 18. The network component of claim 16, wherein aplurality of IP addresses of a plurality of local hosts under each of aplurality of Virtual Local Area Networks (VLANs) are sent in aggregatedform on a periodic basis to update address information in a plurality ofremote gateways in a plurality of remote Layer 2 sites that belong to asingle Layer 2 network.
 19. The network component of claim 16, wherein aplurality of local hosts are grouped to different closed user groups(CUGs) under each of a plurality of Virtual Local Area Networks (VLANs),and wherein the local gateway in the local Layer 2 site sends aplurality of IP addresses under each of the CUGs of each of the VLANs toa plurality of remote gateways in remote Layer 2 sites that belong tothe same Layer 2 network as the local gateway.
 20. The network componentof claim 16, wherein upon receiving a plurality of broadcast AddressResolution Protocol (ARP) or multicast Neighbor Discovery (ND) requests,the local gateway performs a Virtual Local Area Network (VLAN)translation to convert the VLAN of the broadcast ARP or multicast NDrequests to a rooted-multipoint (RMP) VLAN causing all the broadcast ARPand multicast ND requests to be routed to one or more designateddirectory servers.